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Introduction 


This project is broken into two focus areas: telesurgery and surgical rehearsal. In each we are 
exploring various applications and extensions of the existing robotic surgical systems. Under 
telesurgery we are exploring the ability to perform telesurgery using a robot both across a state- 
wide and a nation-wide area based on the currently available technology. Under surgical 
rehearsal exploring designs for simulator systems which can be used to improve training and 
education of surgeons pursuing expertise in the use of robotic surgical systems. The focus is on 
unique forms of robotics which have not previously been addressed by simulation technologies. 


Simulator Performance is an experiment that was part of the scope of the original project which 
began in 2011. We have received permission to complete that study during the time of this 
extension, but using funds remaining from the original project. Therefore, that project is included 
in this report. 


‘The scope described in the statement of work in this report is otherwise limited to the activities 
approved under a funding extension which was provided in September 2014 and extends through 
August 2016. Reports on the original body of work are covered in previous annual reports. 
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Statement of Work 








Telesurgery: Metropolitan Latency. Perform robotic surgical experiments between multiple 
campuses within a metropolitan area, between campuses across a state area, and across 
nationwide campuses, 
Period | Milestone: Telesurgery state-wide latency data and report. Award + 360 days. 
Period 2 Milestone: Telesurgery nationwide latency data and report. Award + 700 days. 








Surgical Rehearsal. Develop virtual reality environment for training operating room staff in 
robotic surgery. Develop design for simulators in hard-tissue robotic surgery (spinal and 
orthopedic). 

Period 1 Milestone: Spinal simulator design document. Award + 300 days. 

Period 2 Milestone: OR team training virtual world environment. Award + 360 days. 

Period 2 Milestone: Orthopedic surgery rehearsal validation report. Award +720 days. 














Project Management 


Progress Summary 
We have received Alpha and Pre-Beta builds of the robotic OR virtual world. Working 
with sub-awardee on revisions and bug fixes. 
+ The data collection for Simulator Performance is completed. We are performing data 
now. 
* We have developed a user analysis for the spinal robot 
*  Telesurgery experiments with Denver are complete. Exploring connection test to Seattle 
rather than Los Angeles. 





simulator. 





Schedule. 
This schedule shows our expected progress and completion of the experiments remaining on this 
grant. 
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Budget. 

Financial spending on the project is under the projected budget. We have added a new MD 
Fellow to the team. We have also used three summer internals who have made significant 
contributions to the progress of the research. 


Scientific Progress 


Simulator Performance. 


Human subject data collection is finished. Expert surgeons were the most challenging population 
to collect. We collected data from Celebration Health, Columbia University Medical Center, and 
at the annual meeting of the Society for Laparoscopic Surgeons. 


We are beginning full analysis of the data from all populations included in the study. The results 
of this work have been accepted for the 2015 ITSEC conference in December; serve as the 
basis for one doctoral dissertation; and are being prepared for journal submission. 


AA basic plot ofthe performance of four different populations while performing the Suture 
Sponge exercise is provide below (lower scores indicate higher skill levels in “economy of 
motion” and “time to complete”), This shows a very distinct performance difference between the 
expert surgeons and all other populations. It also appears that there is little difference between 
the population of lay people, medical students, and video gamers 





es 




















Simulator Design. 


da Vinci OR Virtual World. Through meetings with expert robotic surgeons and their OR staff 
we have arrived at a design for a virtual world which may help surgeons to improve their team 
leadership skills in the OR. Working with ARA/Virtual Heroes Inc. we have created Alpha and 
Pre-Beta builds of a potential virtual world for OR leadership training based on TeamSTEPPS 
principles. The virtual world (or game) will be playable by a surgeon with all other roles being 
played by intelligent avatars. The avatars react to both correct and incorrect actions by the 
surgeon and provide spoken guidance toward the best behavior choices. Screen shots of the 
current build are provided below. 
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A paper on this project has been accepted at the 2015 INTSEC conference. 




















Spinal Robotic Simulator. We are using our knowledge of the Mazor Renaissance spinal robotic 
system to begin to understand how a simulator could be used to improve training for surgeons 
learning the system. Pieces of the surgeon's activities may be represented in: software on a 
laptop, hardware with electronic functionality, and inexpensive replicas of hardware. 





During this quarter we have produced an analysis of the user requirements for such a simulator in 
atraining environment. This is the precursor to a first design for such a device. 





Orthopedic Robotic Simulator. This project will begin in 2016. 
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Telesurger 
Orlando-to-Denver. Communication latency data collection is completed. Average two-way data 


transfer was 72 milliseconds. This speed is much faster than we had expected based on our inter- 
Florida experiments. 





Centura Health Campus, Denver, CO 


Los Angeles vs. Seattle, Original plans were to perform a telesurgery experiment from Orlando- 
to-Los Angeles. The necessary collaboration with that hospital has not materialized. Therefore 
we are now attempting to shift that experiment to a connection test with the University of 
Washington in Seattle. We are also expanding it to include data collection on the performance of 
the Raven Il device. 
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Project WSIXWH-IL 


Key Research Accomplishments 


* Telesurgery: Communications Latency. Major hospital systems have sufficient 
telecommunication bandwidth to perform robotic telesurgery right now. 

‘© Surgical Rehearsal. Simulation-based training for different forms of robotic procedures 
appears to be feasible beyond the simulators of the da Vinci robot which have previously 
been created. We are experimenting with (1) a virtual world for robotic OR team training, 
(2) a simulator to support training on the Mazor Renaissance spinal robotic device, and 
(3) a simulator to support orthopedic robotic procedures on knees and hips. 

‘* Multiple publication and presentation have been generated from this research work. 
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Reportable Outcomes 
Publications 
Mouraviev et al. (Under Review). “Robotic training with porcine models induces less workload 


than virtual reality robotic simulators for urology resident trainees” Submitted to Journal of 
the AUA and AUA Annual Congres 


Smith, Tanaka, Melllwain, Willson. (Dec 2015). “Developing Game-based Leadership Traini 
for Robotic Surgeons.” 2015 Interservice/Industry Training Education and Simulation 
(TSEC) Conference. 


Tanaka, Graddy, Smith, Perez. (Dec 2015). “Gamers Today, Surgeons Tomorrow?” 2015 
Interservice/Industry Training Education and Simulation (INTSEC) Conference. 











Tanaka, Grady, Simpson, Perez, Truong, & Smith. (Accepted). “Robotic Surgery Simulation 
Validity and Usability Comparative Analysis”. Journal of Surgical Endoscopy. 





Tanaka, Perez, Truong, & Smith. “From Design to Conception: An Assessment Device for 
Robotic Surgeons”, 20/4 Interservice/Industry Training Education and Simulation 
(INTSEC) Conference. December 2014. *Best Paper Nominee* 


‘Tanaka, Grady, & Smith. “ Comparison of the Usability of Robotic Surgery Simulators”, 2014 
Interservice/Industry Training Education and Simulation (ITSEC) Conference. December 
2014. *Honorable Mention for Best Paper* 








Smith & Simpson. “ Return on Investment for Robotic Surgical Simulators”, 2014 
Interservice/Industry Training Education and Simulation (IMTSEC) Conference. December 
2014. *Honorable Mention for Best Paper* 


Smith, Truong, & Perez. (2014) Comparative analysis of the functionality of simulators of the da 
Vinci surgical robot. J Surg Endosc, 1-12. 


Perez, Xu, Chauhan, Tanaka, Simpson, Abdul-Muhsin, & Smith. “Impact of delay on 
telesurgical performance: Study on the dV-Trainer robotic simulator”. Submission to Journal 
of Urology 2014. 


‘Smith, “The Future of Robotic Technology 
Press, 2015 (projected) 








”, Robotic Surgery of the Head and Neck, Springer 


Martino, Siddiqui, et al. “Fundamentals of Robotic Gynecologic Surgery” Developing a Quality 
Improvement Project to Improve Patient Safety”, Society of Gynecologic Oncology, Annual 
Meeting on Women’s Cancer, March 2014. 


‘Smith, Patel, & Satava. “Fundamentals of robotic surgery: a course of basic robotic surgery skills 
based upon a 14-society consensus template of outcomes measures and curriculum 
development”, The International Journal of Medical Robotics and Computer Assisted 
Surgery, October 2013. DOL: 10.1002/res.1559 


Smith, “From FLS to FRS: The Fundamentals of Robotic Surgery are on their Way 
Robotic Gynecologic Congress, Chicago, IL. 2013 





World 


Advincula & Smith. “Contributions of Laparoscopic Surgical Experience to the Development of 
Robotic Proficiency”, Society for Gynecologic Surgery Annual Meeting, March 2013. 
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Smith, Chauhan, & Satava. “Fundamentals of Robotic Surgery Consensus: Outcomes Measures 
and Curriculum Development”, NextMed: Medicine Meets Virtual Reality Conference. 
February 2013. 


Smith & Truong. “Robotic Surgical Education with Virtual Simulators”, 2013 
Interservice/Industry Training Education and Simulation (INTSEC) Conference. December 
2013, 


Smith & Chauhan, “Using Simulators to Measure Communication Latency Effects in Robotic 
Telesurgery”, 2012 Interservice/Industry Training Education and Simulation (TSEC) 
Conference. December 2012 *Best Paper Nominee* 





Satava, Smith & Patel. “Report on the First Consensus Conference on the Fundamentals of 
Robotic Surgery” Outcomes Measures”, ACS Accredited Education Institutes Meeting. 
March 2012. 


Presentations 
2015 


Smith (August 2015). “The Validation of Surgical Simulators for RASD”. FDA Workshop on 
Robotically Assisted Surgical Devices. 


Tanaka, Graddy, Perez, Simpson, Truong, Smith. (Nov 2015). “Video Game Impact on Basic 
Robotic Surgical Skills." Annual Meeting of the Association of Gynecologic 
Laparoscopists. 


Perez, Tanaka, Simpson, Truong, Smith, Satava. (Nov 2015). “From concept to surgical 
relevance: Engineering the training device for the Fundamentals of Robotic Surgery.” 
Annual Meeting of the Association of Gynecologic Laparoscoy 


‘Tanaka, Perez, Grady, & Smith. “ Video Game Experience and Basic Robotic Skills”, Florida 
Hospital Internal Research Forum, Orlando, FL, April 2015. 





‘Truong, Tanaka, Simpson, Perez, & Smith. “Robotic surgical simulation versus traditional 
didactics for surgical training: a randomized controlled trial”, Society for Gynecologic 
Surgeons Annual Scientific Meeting, Orlando, FL, March 2015. 


Smith. “Update on Robotic Surgical Simulation”, 2015 Society of Robotic Surgeons (SRS), 
Orlando, FL, February 2015. 








Smith, “Fundamentals of Robotic Surgery”, 2015 Society of Robotic Surgeons (SRS), Orlando, 
FL, February 2015. 


2014 
‘Truong, Tanaka, Simpson, Advincula, & Smith. “A Prospective Randomized Controlled 
Comparative Study on Surgical Training Methods and Impact on Surgical Performance: 


Virtual Reality Robotic Simulation vs. Didactic Lectures”, AAGL Global Congress on 
Minimally Invasive Gynecology, November 2014 


Smith & Simpson. “Return on Investment for Robotic Surgical Simulators”, AAGL Global 
Congress on Minimally Invasive Gynecology, November 2014 





Tanaka, Truong, & Smith. “Robotic Surgical Simulators: An Assessment of Usability and 
Preferences”, AAGL Global Congress on Minimally Invasive Gynecology, November 2014 
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Simpson, Perez, Tanaka, Truong & Smith. “Validating the Efficacy of GEARS through the 
Assessment of 100 Videos”, Society of Laparoendoscopic Surgeons Annual Meeting & 
Endo Expo, September 2014. 


‘Truong, Tanaka, Simpson, Perez, Smith & Advincula, “Randomized Controlled Study 
Comparing Robotic Simulation Versus Didactic Teaching for Robotic Surgical Training: 
Opinions and Perspectives”, Society of Laparoendoscopic Surgeons Annual Meeting & 
Endo Expo, September 2014. *Honorable Mention for the Paul Alan Wetter Award for Best 
MultiSpecialty Scientific Paper* 











Smith & Tanaka. “Gamers in Surgical Simulation: A Comparison of Gamers, Surgeons, and 
Clinical Staff”, Defense GameTech Users Conference, Orlando, FL, September 2014. 


Lendvay, Simpson, Truong, & Smith. “Differentiating Surgical Skill through the Wisdom of 
Crowds”, European Endoscopic Urology Society, April 2014. 


Patel, Patel & Smith, “Feasibility of Robotic Telesurgery across a Multi-Campus Metropolitan 
Hospital System”, Third Biennial Miami Robotics Symposium, April 2014. 


Smith, “Robotic & Telesurgery Research”, Stets 





mn University Senior Tech Expo, March, 2014. 


Satava & Smith, “Fundamentals of Robotic Surgery: Development and Validation of an Online 
Curriculum and New Psychomotor Testing Device”, NextMed/MMVR Conference, 
February, 2014. 


Satava & Smith, “Fundamentals of Robotic Surgery: Development and Validation of an Online 
Curriculum and New Psychomotor Testing Device”, CAMLS-Halldale Summit on New 
Technology in Medicine, February, 2014. 


Tanaka, Truong, Simpson, Perez, & Smith, “A Comparison of the Effectiveness and Usability of 
Robotic Simulators”, Florida Hospital Internal Research Forum, January 2014. 





Smith, “Robotic Surgery Education, Simulation & Telesurgery”, Adventist Health System, 
Surgeon Executives Meeting, January 2014. 


2013 


‘Truong: "The Fundamentals of Robotic Surgery Psychomotor Skills Prototype Development 
Video": Harrith M Hasson Award for Best Presentation Promoting Education and Training, 
2013 SLS Annual Meeting in Reston, Virginia. Smith, “Robotic Surgery Education, 
Simulation & Telesurgery, Society for Laparoscopic Surgeons, Fellowship Summit, 
December 2013. 





‘Smith, “Virtual Reality Simulation: The Future”, Society for Robotic Surgery, Annual Meeting, 
November, 2013. 


Smith, “Strategic Technology Leadership: The Role of the Technology Executiv: 
Leadership Forum, October 2013. 





MITRE 





Smith, “Robots in the Hands of your Surgeon”, IEEE Orlando Chapter Annual Meeting, October 
2013. 


Smith, “Medical Simulation in Robotic Surgery”, Lou Frey Institute of Politics and Government, 
University of Central Florida, September 2013 


Smith, “Robots in the Hands of Your Surgeon”, Chinese American Scholars and Professionals 
Association of Florida, Miami Annual Meeting, August 2013 


Annual Report, September 2015. 
Project W81XWH-11-2-0158 Medical Robotic and Telesurgical Simulation and Education Research 


Smith, “Innovation for Trainers”, Training 2013 Conference. Keynote Presentation, February 
2013. 


2012 





Smith, “Simulation Surgeon, Soldier Spy 
MultiConference. March 2012. 


Keynote presentation, 2012 SpringSim 


Smith, “Robotic Surgery and Surgical Simulation”, presentation to International Council on 
Systems Engineering — Orlando Chapter. February 2012. 


Smith, “Beyond Education and Training: Challenges of Running Medical Simulators in New 
Paradigms”. 202 International Meeting on Simulation in Healthcare. January 2012. 

2011 

Smith, "Simulation in Surgical Education”, American College of Healthcare Executives, 
December 2011. 


Smith, "Medical Simulation Special Event: Robotic and Telesurgery Research Using 
mulation", /TTSEC, December 2011. 





Smith, "Robotic and Telesurgery Research", National Center for Simulation, October 201 1. 





Smith, "Medical Simulation Standards: What can we lean from the DoD?" 
Training, and Treatment Conference, May 2011 


Medical Technology, 


Smith, "Simulation and Game Technology in Medical Education", IDEAS Workshop, Harvard 
Medical School, April 2011. 


‘Smith, "Robotic Surgery and Surgical Simulation", Guest Lecture, Old Dominion University, 
April 2011 


2010 
Smith, "Surgical Simulation Research Initiatives", /ITSEC UCF Workshop, December 2010. 
Smith, "da Vinci Surgical Robot", /ITSEC techPATH Teachers Workshop, November 2010. 


Poster Presentations 
2015 


Smith, Simpson. “Return on Investment Model for Robotic Simulators”, Poster Presentation at 
2015 Society of Robotic Surgeons (SRS), Orlando, FL, February 2015. 

‘Tanaka, Graddy, Abdul-Muhsin, Simpson, Truong, & Smith. “A Comparison of Validity and 
Usability of Robotic Simulation”, Poster Presentation at 2015 Society of Robotic Surgeons 
(SRS), Orlando, FL, February 2015. 

Tanaka, Perez, & Smith. “ Fundamentals of Robotic Surgery Psychomotor Skills: Metrics 
Development and Evaluation”, Poster Presentation at 20/5 International Meeting on 
Simulation in Healthcare (IMSH) Conference, New Orleans, LA, January 2015. 


2014 


Lendvay TS, White LW, Holst D, Kowalewski T, Harper JD, Sorenson M, Brand TC, Truong M, 
Simpson K, Smith R. Quantifying Surgical Skill Using the Wisdom of Crowds. American 
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College of Surgeons Clinical Congress, San Francisco, CA, October 26-30", 2014. [Poster 
#PP2014-51161] 


Lendvay T, Holst D, White L, Kowalewski T, Brand T, Sorenson M, Harper J, Truong M, 
Simpson K, Smith R. Differentiating Surgical Skill Through the Wisdom of Crowds. 
American Urological Association Annual Meeting, Engineers in Urology Session, Orlando, 
FL, May 16-21, 2014 [Moderated Poster #82] 
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Awards 
Nominee, 2015 Florida Hospital Des Cummings Innovator Award 


Best Paper Nominee in the Emerging Concepts and Innovative Technologies track, 2014 
Interservice/Industry Training Education and Simulation (I/ITSEC) Conference, Orlando, FL. 


Honorable Mention for Best Paper in the Training track 20/4 Interservice/Industry Training 
Education and Simulation (WITSEC) Conference 


Honorable Mention for Best Paper in the Policy, Standards, Management and Acquisition track 
2014 Interservice/Industry Training Education and Simulation (IITSEC) Conference, Orlando, 
FL. 


2014 Florida Hospital Des Cummings Innovator Award 


Best Paper Nominee in the Human Performance track 2014 Interse 
Education and Simulation (INTSEC) Conference 





iice/Industry Training 


2014 Honorable Mention for the Paul Alan Wetter Award for Best Multispecialty Scientific 
Paper, MIS Week Annual Conference 


2013 Second Place, Top Gun Surgery Competition, MIS Week Annual Conference 


2013 Harrith M. Hasson Award for Best Presentation Promoting Education and Training, MIS 
Week Annual Conference 


2013 Best Video Session in Multispecialty Surgery, MIS Week Annual Conference 
2013 Silver Medal, Robotic Surgery Olympics, MIS Week Annual Conference 
2013 Third Place, Top Gun Surgery Competition, MIS Week Annual Conference 


2012 Schwartz Industry Innovation Award, Orlando Economic Development Commission. The 
Nicholson Center's research work in robotic telesurgery was recognized locally as one of the 
most innovative activities in the Orlando metropolitan area. 


2012 Best Paper Award for Simulation Technologies Track, Interservice/Industry Training 
Education and Simulation Conference (I/ITSEC) 
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Conclusion 
Each of the research areas funded by this grant has made significant scientific contributions. The 
knowledge gained from this work is being shared through reports to the government and multiple 
presentations at both clinical and simulation conferences. We have also submitted multiple 
papers for journal publication. 


‘This cooperative agreement is scheduled to end on August 31, 2016. Based on our current work 
flow and state of funds the project is currently on schedule to complete all objectives by the end 
of the agreement. 
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Appendices 


Copies of manuscripts, abstracts, and presentations of work resulting from this grant are included 
as appendices to this report. 
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Surgical Endoscopy 


Dr. Mark A. Talamini 
Editor-In-Chief 
March 5, 2015 


Dear Editor, 


Please find enclosed an article we wish to submit for publication in The Journal of 
Surgical Endoscopy entitled: “Robotic Surgery Simulation Validity and Usability 
‘Comparative Analysis.” Our team has been conducting work relating to minimally 
invasive surgery for many years. We seek publication in The Journal of Surgical 
Endoscopy because it provides the surgical community a juncture to exchange critical 
information on practice, theory, and research in various medical and surgical disciplines. 
The evaluation of the usability and validity of available training tools is valuable and 
relevant information to in the surgical community. Also, this paper details the second 
phase of work previous published in the journal entitled “Comparative analysis of the 
functionality of simulators of the da Vinci surgical robot.” 














This manuscript has not been published and is not being considered for publication 
elsewhere, There are no financial or other relations that could lead to a conflict of 
interest. Each author has contributed significantly to the submitted work: 

1) Conception and Design: Roger Smith, Mirelle Truong. 

2) Data acquisition: Alyssa Tanaka, Courtney Graddy 

3) Data Analysis and interpretation: Alyssa Tanaka, Khara Simpson, Courtney Grady 
4) Drafting the manuscript: Alyssa Tanaka, Courtney Graddy, Khara Simpson, Manuela 
Perez 

5) Critical Review of the Manuscript: all authors 

6) Supervision: Roger Smith 


All authors have read and approved the final version of this manuscript. 
The address for correspondence is: 


Alyssa Tanaka, 

404 Celebration PL. 

Celebration, FL 34747 

Phone: (407) 303-4276; Mobile: (321) 480-5510; E-mail: Alyssa.tanaka@flhosp.org 


We appreciate your time and consideration. 


Respectfully, 
The authors, 


worvourens 
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Robotic Surgery Simulation Validity and Usability Comparative Analysis 


Running Head: Robotic simulation validity and usability 


Authors: 
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Courtney Graddy, MHA, Florida Hospital Celebration Health, Celebration, Florida 
Khara Simpson, MD, Columbia University Medical Center, New York, NY 
Manuela Perez, MD, PhD, University of Lorraine-Nancy, Nancy, FR 

Mireille Truong, MD, Columbia University Medical Center, New York, NY 
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Surgical Endoscopy 


ABSTRACT 


Background: The introduction of simulation into minimally invasive robotic surgery is relatively recent and has seen 
rapid advancement; therefore, a need exists to develop training curriculums and identify systems that will be most 
effective at training surgical Skills. Several simulators have been introduced to support these aims -- the daVinci 
Skills Simulator, Mimic dV-Trainer, Surgical Simulated Systems’ RoSS, and Simbionix Robotix Mentor. While 
multiple studies have been conducted to demonstrate the validity of these systems, studies comparing the perceived 
value of these devices as tools for education and skills are lacking. 





Methods: Subjects who qualified as medical students or physicians (n=105) were assigned a specific order to use 
each of the three simulators. After completing a demographic questionnaire, participants performed one exercise on 
the three simulators and completed a second questionnaire regarding their experience with the device. Afer using all 
systems, they completed a final questionnaire, which detailed their comparative preferences. The subject’s 
performance metrics were also collected from each simulator. 


‘Results: The data confirmed the face, content, and construct validity for the dV-Trainer and Skills Simulator, Similar 
validities could not be confirmed for the RoSS. Greater than 80% of the time, participants chose the Skills Simulator 
in terms of physical comfort, ergonomics, and overall choice. However, only 55% thought the skills simulator was 
‘worth the cost of the equipment, The dV-Trainer had the highest cost preference scores with 71% percent of 
respondents feeling it was worth the investment. 





Conclusions: Usability can affect the consistency and commitment of users of robotic surgical simulators. In a 
previous study, these simulators were objectively reviewed and compared in terms of theit system capabilities. 
Collectively, this work will offer end users and potential buyers a comparison of the perceived value and preferences 
of robotic simulators. 


KEYWORDS, 


Simulation; Validation; Robotic Surgery; Training; Usability 


Surgical Endoscopy 


INTRODUCTION 


Medicine has come to the conclusion that the Halstedian training model (ie., See one, do one, teach one) is no 
longer sufficient for teaching complex skills, particularly robotic surgical skills [1]. With the introduction of robotic 
technology between patient and surgeon, a need to master new skills has emerged. A number of virtual reality 
simulators have been developed to support the training and acquisition of such skills. Currently, the commercially 
available robotic simulators include: the da Vinci Skills Simulator (4VSS) by Intuitive Surgical Inc., also known as 
the "Backpack Simulator”; the dV-Trainer from Mimic Technologies Inc.; the RoSS by Simulated Surgical Sciences 
LLC; and the Robotix Mentor from Simbionix (Figure 1). All of these da Vinei simulators utilize a visual scene that 
is presented in a computer-generated 3D environment providing challenging tests for practicing dexterity and 
machine operations. Originally, the simulated exercises trained basic robotic skills; however with advances in 
technology, surgeons can now train for specific procedures (e.g. partial nephrectomy and hysterectomy), 








Figure 1, Simulators of the da Vinei robotic surgical system 


‘The work described in this paper is the second part of a three-phase analysis to study the effectiveness of these 
simulators and applications to the education of robotic surgeons. In the first phase, the authors evaluated and 
compared the objective characteristics of three simulators (4VSS, dV-Trainer, and RoSS), The Simbionix Robotix 
Mentor was not included because it was under development at the time of this research. This analysis provided a 
head-to-head comparison of the systems and found that they varied greatly in their hardware and software, 


In the dVSS, the trainee operates the simulated environment using the actual da Vinei surgical console. The 
simulator is a custom computer, appended to the surgical console through the surgical data port. While the simulator 
costs approximately $85,000, the surgical console costs $500,000 incurring an investment of $585,000. Using this 
simulator, users can train with the actual hardware they would use during surgery; however, this requires availability 
‘of the surgical console, which may be fully scheduled in the operating room. Few hospitals have a dedicated training 
console, meaning that users do not have ready access to the simulator, The second system is a standalone system that 
utilizes a high performance graphic/gaming computer, connected to a custom desktop viewing and control device 
that replicates the hardware of the da Vinci surgeon's console. This system shares similar software with the dVSS, 
bbut does not require the use of actual da Vinei hardware. The cost of this simulator is approximately $96,000. The 
third system is composed of a completely customized replica of the da Vinci surgeon's console. Internally the 
simulator contains a graphic computer, a 3D viewing system, and commercial Omni Phantom haptic controllers. 
This simulator uses unique software and costs approximately $126,000 [2] 





‘The validity of medical and surgical simulators is typically evaluated using the categories defined by McDougal [3]. 
‘This paper defines the most commonly recognized forms of validation as: face, content, consiruct, concurrent, and 
predictive validity. Face validity is typically assessed informally by users and indicates whether the simulator is an 
accurate representation of the actual system (ie. the realism of the simulator). Content validity is the measure of the 
appropriateness of the system as a teaching modality. Experts who are knowledgeable about the device typically 
assess this via a formal evaluation. Construct validity is the ability of a simulator to differentiate between the 
performances of experienced users and those who are novices. Concurrent validity is the extent to which the 
simulator correlates with the “gold standard” for training and predictive validity is the extent to which the simulator 
ccan predict a user’s future surgical performance. Collectively, concurrent and predictive validity are known as 
criterion validity and are used as measures of the simulator’s ability to correlate trainee performance with their real 
life performance. Face and content validity are most effective in evaluating the ability of a simulator to train a 
surgeon; however construct, concurrent, and predictive validity are most useful for evaluating the effectiveness of a 
simulator to assess a trainee. 





‘The validity of all three simulators has been examined separately (Table 1) and to our knowledge there is no 
‘comparative research of all three systems. The current study therefore compares the three commercially available da 
Vinci simulators and details the findings for face, content, and construct validity of these systems. The purpose of 
this is to provide end-users and potential buyers with a head-to-head evaluation of the value and usability of the 
systems, 
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‘Table 1. da Vinei simulator validation studies from Smith R, Truong M, & Perez M [2] 
MATERIALS AND METHODS, 


Participants in this study included medical students, residents, fellows, and attending physicians. Participants were 
recruited from the University of Central Florida College of Medicine, courses held at the Florida Hospital Nicholson 
Center, and two surgical robotics conferences (World Robotics Gynecology Congress and Society of Robotic 
‘Surgeons Scientific Meeting). Subjects were excluded from participating if they had participated in a formal robotic 
simulation-training course to eliminate preference bias, Each participant was categorized into one of three groups 
(ie. Expert, Intermediate, or Novice) according to the self-reported number of robotic cases performed. Individuals 
‘who had performed 0-19 robotic cases, were categorized as Novices, individuals with 20-99 robotic cases were 
considered to be Intermediates, and individuals with 100 or more cases were considered to be Experts. 


‘After being categorized into an experience level, each participant was assigned a specific order in which they used 
each of the simulators (Figure 2). This alternating order was implemented to identify and eliminate any potential 
bias that may exist by using a specific system first. All participants completed one exercise on each of the 
simulators. The tasks chosen were Peg Board | in both the dV-Trainer and the dVSS and Ball Placement | in the 
RoSS. The same task was used for both the dV-Trainer and the dVSS because these systems share similar software 
and exercises. The RoSS software contains unique exercises and Ball Placement | was chosen because it tains the 
same basic skills as Peg Board I 


Figure 2. Example of rotating order and research process 


‘After completing the exercise on a simulator, participants completed a post-questionnaire (Survey 1), which asked 
for feedback regarding their experience on that specific simulator. After using all three systems, subjects completed 
2 second post-questionnaire (Survey 2), which asked them to compare all three systems to each other. The 
participant's performance metries were also collected from each of the simulators. 


RESULTS 





The Novice group (n=37) had performed an average of 2 robotic cases, the Intermediate group (n=31) on average 
performed 54 cases, and the Expert group (n=37) performed 336 eases. Sixty-two percent of subjects were men and 
38% were women with an average age of 43. On average, participants had 15 years in practice and 3 years of robotic 
‘experience. Seventy-six percent were attending physicians and 73% of participants were currently or had received 
robotic surgery training, while 41% provided that they train residents and fellows. A one-way ANOVA verified a 
difference in the average age and number of years in practice of participants based on the classification of expert, 
intermediate or novice (number of robotic procedures). This is to be expected since higher ages typically imply a 
higher number of years of practice and resultant larger numbers of robotic procedures, 


‘The types of validity evaluated in this experiment were face, content, and construct. To analyze the systems for face 
validity and content validity, questions fom Survey 1 were used. The questions were evaluated on a five point 
Likert scale (Le., Strongly Disagree, Disagree, Neither Agree or Disagree, Agree, and Strongly Agree). As 
recommended by Van Nortwick et al. [26], face validity was analyzed by expert and intermediate feedback only as 
these are the users most familiar with the robotic system: however, only expert feedback was used for content 
validity because they have the best ability to judge the appropriateness of the system as a training tool. For construct 
validity, performance metrics such as Overall Score, Time to Complete, Number of Errors, and Economy of Motion 
‘were analyzed (Table 2). Specifically, Time and Economy of Motion were chosen due to a previous study by 
Perrenot, Perez, Tran, Jehl, Felblinger, Bresler, & Hubert [10] indicating that these are highly relevant indicators of 
‘expertise in robotic surgery, 





Table 2. Description of data used for types of validity. 


Face Validity 
‘A Chi-square test of independence was used to evaluate the distribution of scores for a specific simulator in relation 
to the order of the system's presentation to the subject. This analysis indicated that there was no difference in 
participants’ responses according to the order in which the systems were presented: and established that no bias was 
present due to the presentation order (p>0,05). These questions asked participants to evaluate whether the hand 
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controllers on the simulator were effective for working inthe simulated environment (Question 1) and ifthe device 
isa sufficiently accurate representation of the teal robotic system (Question 4). For both questions, the RoSS had the 
lowest average score, dV-Trainer had the second highest score, and the dVSS had the highest score of the three 
(Cable 3). A repeated measures ANOVA verified that the answers were statistically different for both questions 
(p-00001), 


‘Table 3. Mean scores from a 5-point Likert seale on face validity 


Content Validity 
‘As seon in Tale 4 100% of participants iter agreed or strongly agreed that the 3D graphical exercises inthe 
AVSS were effective for leaching Fobotic sis while 39% disagreed or swongly disagreed that the, RSS" 
capabilites were effective. When asked ifthe seorig stem effectively communicated their performance. 88% of 
VSS users agreed or strongly agreed, while 79% of €V-Trainer users agreed or strongly agree, Similarly, 91% and 
82% of participants agreed or songly agreed that the dVSS and dV-Tainer, respectively effectively guided them 
to improve their performance, while only 36% felt the RoSS provided the sume guidance, 








Table 4. Percentages of Likert responses for content validity questions 





Construct Validity 
‘The overall score, number of errors, time to complete, and economy of motion scores collected by the simulators for 
Experts (n=37) and Novices (n=37) were used to compare construct validity (Table 5). Intermediate subjects were 
not included in the construct validity analysis because it was only necessary to determine if the simulator could 
distinguish specifically between novice and expert users. Overall Score is synthesized from multiple metrics and is 
specific to the individual simulator. This metric was available in the dVSS and the dV-Trainer, however the Overall 
Score metric is not automatically exported by the RoSS and therefore was not analyzed for this system. Instead, the 
Number of Errors was used for the RoSS. For all of the simulators, higher Overall Score values are better, while 
lower Economy of Motion, Time, and Number of Error values are better preferred, 





For the RoSS, the analysis has 23 missing data points because the system does not report scores when a user exceeds 
‘a maximum exercise time or chooses to terminate the exercise before completion. This resulted in a sample of 30 
experts and 21 novices on this system. A Mann-Whitney U test showed that the distributions of time (p=0.221), 
number of errors (p=0.644), and economy of motion (p=0.566) were not statistically different for the experts 
‘compared to the novice group on this simulator. 











The dV-Trainer analysis of experts (n=37) and novices (n=37) had three missing values for economy of motion and 
completion time and five for the overall score metric, thus the analysis contained varying number of subjects. The 
distribution of the overall scores was not significantly different for the expert compared to the novice group 
(p-0.061). These tests did confirm statistical differences for economy of motion (p<0.001) and time to complete 
(p<0.001), with a lower economy of motion value and shorter completion time for experts compared to novices. 











The dVSS analysis included all novice (n=37) and expert (n=37) participants. Time to complete (p0.001) and 
‘overall score (p=0.006) were significantly different for the expert compared to the novice group. The expert group 
had a higher overall score and a shorter completion time compared to the novice group. However, economy of 
motion did not show a statistical difference with this analysis (p=0.216) 





Table 5. 





lann-Whitney U test level of significance on construct validity measures 


‘The relationship between experience and performance metrics was more specifically analyzed in terms of the self 
reported number of cases of all participants (n=105) using a non-parametric correlation coefficient (Spearman's). 
For the RoSS, 30 participants were excluded from the analysis. For the participants that were included in the 
analysis (n=75), there was not a significant correlation between Time to Complete (p=0.181), Number of Errors 
(p=0.563), or Economy of Motion (p=0.390) with the total number of robotic cases performed (Figure 3), 





Figure 3. Graphs of correlation between experience and metrics on the RoSS 
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For the dV-Trainer, four participants were excluded from the entire analysis and two participants were excluded 
from the Overall Score analysis (Overall Score n=99; Economy of Motion and Time to Complete n=101), The 
analysis verified a statistically significant correlation between Overall Score (p=0.03), Economy of Motion 
(p<0.01), and Time to Complete (p<0.01). The correlation value was negative for Economy of Motion and Time to 
Complete, showing that with a greater number of robotic cases, the time taken and distance moved decreased. The 
correlation was positive for Overall Score indicating that the participants’ score increased with the number of 
robotic cases performed (Figure 4). 





Figure 4, Graphs of correlation between experience and metrics on the dV-Trainer 


For the dVSS, two participants were excluded from the analysis (n=103). A statistically significant difference was 
found between Overall Score (p =0.01) and Time to Complete (p <0.01). The correlation value was negative for 
time and positive for Overall Score, signifying that with more robotic cases the time taken decreased and the score 
increased. There was not a statistically significant correlation between Economy of Motion and the total number of 
robotic cases performed (p=0.105) (Figure 5). 





Figure 5. Graphs of correlation between experience and metries on the dVSS 
Usability (Preference) 


The questions from Survey 2 were used to understand the preference of the subjects when using the simulators. All 
subjects were included in this analysis except for two participants who were dropped from the analysis because they 
did not complete the questionnaire. The participants’ responses to the following usability questions can be seen in 
Figure 6: 





Ifyou are (were) a program director, which simulator would you choose for your trainees; 
In which simulator were you physically more comfortable; 

Which simulator had the best hand controls: 

Which simulator had the best foor controls; 

Which simulator had the best 3D vision, 

Were you feeling stressed or annoyed by any of the simulators? 


Figure 6. Description of usability responses 


Overall, most participants preferred the dVSS and indicated that they would choose this device as a taining system 
if they were a program director, Participants not only felt most comfortable in the dVSS, but also felt that the system 
had the best control and vision equipment. The least preferred system was the RoSS, which most participants also 
agreed made them feel stressed or annoyed. Ten percent of participants also responded that they felt stressed or 
annoyed by both the dV-Trainer (dVT) and the RoSS. 


Cost 


All participants were also asked to provide feedback on their simulator preference in terms of the cost of the system, 
The responses were analyzed in terms of the frequency of the responses given. Most participants felt that the dV- 
Trainer was worth the investment; while most felt that the RoSS was not. When asked about the dVSS, only 56% of 
participants agreed that it was worth the investment (Figure 7). 


Figure 7. Description of cost preferences, 
DISCUSSION 


The aim of this study was to conduct a comparison of the three commercially available simulators used to train 
surgeons on the daVinci robotic surgical system. The study was performed to assist potential buyers in making a 
purchasing and deployment decision regarding robotic simulators. This study provides information about the face, 
content, and construct validity, as well as usability of the systems. 
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‘The simulators were perceived to be different in their representation of the real robotic system, The dVSS was most 
preferred in terms of ergonomics and usability; however, most participants did not feel that this system was worth 
the investment. The costs provided in the questionnaire included all equipment needed to make the simulator 
functional. While the simulator itself only costs $85,000, it is impossible to use without the $500,000 da Vinei 
surgeon console. By leveraging the actual da Vinci hardware, this simulator allows for a more realistic experience, 
but limits the availablity and creates a higher cost for training than other robotic simulators. Economy of Motion 
‘was not able to differentiate novices from experts in the dVSS, which could be attributed to the ease of use of the 
controllers allowing novices to move the controls as efficiently as experts. The generous workspace of the dVSS 
could also have an impact on the lack of difference. 


In terms of cost, most participants agreed that the dV-Trainer had the best cost-effectiveness. In contrast to the 
VSS, the dV-Trainer isa standalone simulator and does not require the support of the daVinei hardware to operate. 
This allows for better accessibility and requires less of an investment for training. The Overall Score aspect of 
construct validity in the dV-Trainer may not have shown a difference between novices and experts due to the way 
that the scoring is developed. The scoring system is constructed with a “ceiling” that prevents users from achieving a 
high Overall Score without attaining high scores across multiple metres. 


‘The RoSS was the least preferred system for comfort and other usability aspects (ie., hand controls, foot controls, 
and 3D interface), with most participants feeling stressed or annoyed when using the system. This study was unable 
to validate the face, content, or construct validity for this system. Currently, there is limited data available that 
confirms construct validity of the RoSS. Similarly to Raza [21], this study was unable to confirm a difference 
between experts and novices in terms of time taken to complete the exercise. AAs stated previously, time and 
‘economy of motion are considered highly relevant measures of expertise levels [10] and should distinguish between 
these groups in the simulators, 


‘To our knowledge this three-part study is the first to compare three of the available simulators. This study involved 
the largest sample size and diversity of participants (ie., experience levels, number of robotic cases, and 
subspecialty type) thus far in relevant publications. The results from this research will help guide the choice of 
simulators used for future studies at Florida Hospital and may also influence decisions at other laboratories. 
However, a limitation to the study was the lack of consistency in the available exercises and scoring systems across 
the three systems. A consideration for future studies will be to use more complex exercises and increase the depth of 
the face and content validity evaluation. Future research is also necessary to evaluate and compare new iterations of 
dda Vinei simulators (e.g the Simbionix Robotix Mentor). 
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Figure 3. Graphs of correlation between experience and metrics on the RoSS 
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Figure 5. Graphs of correlation between experience and metrics on the dvSS 
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Type of | Evaluation Type of ‘Question/Met 
Validity Participant 
Face Suney1 | Expert and | Ql; The hand controllers on this simulator are elfective Tor working 
Validity Intermediate in the simulated environment (Liker). 
(Qa: The device is a sullicienily accurate representation ofthe real 
robotic system (Likert) 
Content | Survey 1 | Expert (Qi: The 3D graphical exercises in the simulator are elfective Tor 
Validity teaching robotic skills (Likert. 
‘QS: The scoring system effectively communicates my performance 
on the exercise (Likert) 
‘Qe: The scoring system effectively guides me to Improve 
performance on the simulator (Likert). 
Construct |~ Simulator | Experts and | Overall Score (points) 
Validity Novices Number of Errors (count) 











“Time to Complete (Seconds) 








Economy of Motion (centimeters) 
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‘Table 3. Mean scores from a 5-point Likert scale on face validi 
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(Qa: The device is a sufficiently accurate representation of | 4.65 3 TR 
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Table 4, Percentages of Likert responses for content validity questions 





‘Content Validity (a=34) 
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10 Table 5. Mann-Whitney U test level of significance on construct validity measures 
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GEARS Study 
SLS Abstract 
Validating the Efficacy of GEARS through the Assessment of 100 Videos 


Objective: To evaluate the use of the GEARS assessment to distinguish varying robotic surgical 
experience. To identify additional variables that may increase the reliability and accuracy of GEARS. 


Methods: Videos were collected for 104 medical students and surgeons performing a simple cystotomy 
closure on an animate model. Subjects were divided into three groups based on their robotic 
experience. Reviews were performed by three surgeons (inter-rater reliability 0.95 cronbach’s alpha) 
Using GEARS, errors and tasks assessments. Statistical analysis was performed to determine the 
validity and internal consistency of GEARS as well as its correlation with the independently developed 
errors and tasks check lists. 


Results: There were differences between task time, GEARS, error and task metrics across experience 
groups (p = 0.01). For the individual categories of GEARS, all were able to differentiate between 
experience groups. Bimanual dexterity and efficiency were the best at differentiating intermediates (26- 





100 robotic cases) from experts (+100). In the evaluation of errors, missed targets, needle drops, tissue 
damage, and instrument collisions were statistically different among groups and instruments out of 
view, needle re-loading, tissue re-grasping were not. Instrument collision and needle drops showed the 
greatest differentiation among groups. GEARS had internal consistency (cronbach’s alpha 0.88) and 
construct validity (p=<0.001). 





Conclusions: This is the first study to confirm the ability of GEARS to differentiate robotic surgical 
experience at all levels of training. While errors and task assessment also showed differences among 
experience levels, they were less reliable than GEARS. 


INTRODUCTION: 


The inception of robotic surgery has changed the face of surgery by providing an additional means to 
completing more complex procedures in a min. invasive fashion. While there are obvious parallels with 
traditional laparoscopic surgery, endowrist manipulation and lack of haptic feedback represent 
significant differences. Due to its unique interface, robotics has necessitated device specific training and 
evaluation. To this aim, several robotic surgical simulators have been developed and validated. The 
Fundamentals of Robotic Surgery (FRS) is also being trialed and validated to offer specialty non-specific 
didactic and simulation training, including a high stakes test. 


For objective in-OR and video assessment, the global evaluative assessment of robotic skills (GEARS) is 
used most often. GEARS, similar to GOALS (a validated laparoscopic objective assessment tool), is a six- 
item objective assessment using a Likert scale. The six fundamental skills assessed are: depth 
perception, bimanual dexterity, tissue handling, efficiency, autonomy, and robotic control. The first five 


skills are the same as GOALS. The five point likert scale has definitions associated with 3 of the 5 scores. 
GEARS has been found to have construct validity, and good intra and inter-rater reliability (Goh et al 
2012). No studies have been done evaluating its ability to differentiate levels of expertise and which 
skills sets/categories best correlate to proficiency/surgeon expertise level. We propose to evaluate the 
above as well as to test additional structured objective assessments (error based and task based scoring) 
to better categorize GEARS. 


Our hypothesis is that collisions and tissue damage will be highly associated with novice performance 
and instrument out of view will be a less discriminatory factor for experts. 


BACKGROUND 


GOALS was created from the concept of OSATS which includes the objective assessment of skills and a 
task assessment. 


METHODS 


Archived videos and assessments were reviewed for 100 surgeons performing a simple cystotomy 
closure. Previous assessements included the GEARS and an addendum questionnaire regarding errors 
and task assessment. The errors were designed to correlate with each of the GEARS items. (Look 
below). Reviews were performed by three independent reviewers whose interrater reliability was 95 
via cronbach’s alpha. The trained reviewers were all familiar with robotic surgery and included two 
minimally invasive fellows and an attending surgeon. Training consisted of reviewing the GEARS, 
encouraging full use of the scale, and a focus on documentation of only confirmed errors. Reviewers 
were blinded to the experience level of the surgeons. Intra questionnaire correlation was performed 
Within the GEARS as well as with the addendum questionnaire to determine internal consistency. An ICC 
was performed for the total GEARs score as well as with the deletion of each item. A similar process was 
performed for the task and error portion of the questionnaire. Finally the overall scores of each (GEARS, 
task, error) were compared to see if there was a relationship. The mean and median scores were 
calculated for each based on experience levels. Compare each item with experience level to determine 
Which item most correlated with the three experience categories. Scatterplot graphs were performed 
comparing the number of robotic surgeries peformed with the total GEARS scores, errors, and tasks. 
Confirm that the developed errors and task questions correlated with the initially identified items in 
GEARS. 


GENERAL ANALYSIS 


1. Mean and median score for each experience group for the GEARS, errors, and task assessment. 
Determine if these differences are statistically different. HYPOTHESIS: there will not be a 
statistically significant difference between intermediates and experts but it will be present 
between novices and experts. RESULTS ~ There were statistically significant differences across 
all experience levels. 


2. Identify which components of the GEARS or errors or task assessments where there are 
statistically significant differences — item by item by analysis. 


3. Looking at correlation between the items with no statistical significance to determine if 
associated task or error components gave better differentiation between the two groups. 


SUB-ANALYSIS 


1. Identify relationships between the associations between GEARS items and the errors and task 
(see below) HYPOTHESIS: 


2. Come up with a total score/equation including the GEARS, errors (neg scoring), and 
tasks....compare that to the different experience levels and see if there is a statistically 
significant difference among the groups 


RESULTS 
CONCLUSION/DISCUSSION 


Considerations — We did not fully evaluate the benchmark of autonomy. As noted above subjects were 
given the same task to complete without assistance. Reviewers in most cases gave 


LIT REVIEW: Google scholar search term GEARS 
GEARS: Original study 
Jan 2012 Goh et al - Journal of Urology 


- Used expert consensus to identify an objective assessment tool for the eval of robotic surgeons — 
GOALS with an additional variable for robotic control 


- Determined to have good internal consistency and reliability. Also established construct validity but 
interestingly enough could not differentiate PGY-6 (avg surgical cases 30) from expert level surgeons 
(avg surgical cases 190) 


~ 4 attg surgeons, 25 trainees ( PGY4-6) 
‘Teaching surgical skills ~ Reznick NEJM 2006 
Reviews changes in medical education and considerations for the future 


Traditional halstedian model of education stems from education through volume of patients. With 
current advances in medicine the people are living longer, increasing the complexity of conditions. 
Certain conditions are becoming increasingly more rare as well. We also have to contend with work hour 
shortages. To improve/strengthen training and exposure ~ we are turning to simulation for orientation 
and to improve outcomes. 


Learning Tools and simulation in Robotic Surgery: State of the Art ~ Citation Pubmed 


-Fitts and Posner's 3 stage theory of motor skill acquisition ~ suggests that in the first phase the learner 
is more focused on the steps of the material and typically is less coordinated. This suggests that maybe 
this stage of learning should be performed in a non-lethal environment 


Perform searches on FLS/OSATS/Goals/Include RTN 


Conclusions - (Discussion) Unable to differentiate between intermediate and expert level surgeons with 
GEARS alone. Need another variable or term for surgical skills. 


-Limitations ~ Did not fully assess autonomy ~ everyone defaulted to a S. 
Meeting Notes 4/28/14 
Correlations between GEARS and errors 

1. Depth perception - Intstrumentation 

2. Bimanual dexterity ~ Needle Handling and 

3. Tissue Handling Tissue handling 

4. Efficiency Time 

5. Autonomy — Nothing/didn’'t fully evaluate 


6. Robotic Skills ~ Instrumentation and needle handling and tasks 


Current status of robotic simulators in acquisition 
of robotic surgical skills 





Anup Kumar", Roger Smith”, and Vipul R. Patef* 





Purpose of review 
This article provides an overview ofthe current status of simulator systems in robotic surgery training 
curriculum, focusing on available simulators for training, their comparison, new technologies introduced in 
simulation focusing on concepts of training along with existing challenges and future perspectives of 
simulator taining in robotic surgery. 


Recent findings 

‘The different virlual reality simulators available in the market like dVSS, dVT, RoSS, ProMIS and SEP have 
shown face, content and consiruct validity in robotic skills traning for novices outside the operating room. 
Recenily, augmented reality simulators like HoST, Maestro AR and RoboliX Mentor have been introduced in 
robotic training providing a more realistic operating environment, emphasizing more on procedurespecific 
robotic raining . Further, the Xperience Team Trainer, which provides training to console surgeon and bed- 
side assistant simultaneously, has been recently inlroduced to emphasize the importance of leamwork and 
proper coordination. 

Summary 

Simulator training holds an important place in current robotic training curriculum of future robotic surgeons. 
There is a need for more procedurespecific augmented realty simulator training, uilizing advancements in 
computing and graphical capabilities for new innovations in simulator technology. Further studies are 
required to establish its cost-benefit ratio along with concurrent and predictive validity. 

Keywords 

robotics surgery, simulation, surgical training, virtual reality 








INTRODUCTION 
The use of the robotic platform in urology has 


anatomy and an understanding of magnification, 
depth perception, three-dimensional spatial orien- 


expanded exponentially over the last decade and 
has established itself in most advanced centres across 
the world, particularly in the USA [1-3]. In 2013, 
approximately 80% of radical prostatectomies were 
performed using robotic platform in the USA [1]. This 
tremendous growth in robotic technology has high- 
lighted the increasing demand for surgeons trained 
{in robotic skills. Although most urology residency 
programs are presently incorporating roboticsurgery 
as a part of their curriculum, adequate training of 
these future robotic surgeons is facing many chal- 
lenges [4-6]. First, there has been a decrease in actual 
training hours along with risk of litigation, increased 
emphasis on patient safety and improved surgical 
outcomes. Second, the traditional Halstedian 
method of training of ‘see one, do one and teach 
one’ does not apply to robotic technology. The 
robot-assisted radical prostatectomy is a complex 
procedure requiring complete knowledge of pelvic 
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tation and coordinated hand-eye movements. 
‘Third, in robotics, the mentor is not working close 
to the trainee with one person at the console and one 
other person required for bedside assistance, thus 
raising concerns in the mentor’s mind about the 
patient's safety [7-9]. The training can be divided 
aspreclinical and clinical [4-6]. The preclinical train- 
ing includes use of simulators, defined as tools 
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KEY POINTS 


+ The simulator ining can form an negra pat of 
credentialing and training robotic surgery of future 
robotic surgeons. 


«= Ithas the potential to decrease the learning curve for 
the acquisition of robotic sil, 


« It.can supplement the hands-on training clinical phase 
and can act as a bridge between preclinical training 
‘and actual hands-on clinical training without 
jeopardizing the safety of patients. 

«= There is @ need for more procedurespecific augmented 
techy sinultor raining ina costeffacive manne, with 
‘more emphasis on both technical skis and team- 

‘work raining, 


enabling the operator to reproduce or represent 
under test conditions a phenomenon likely to occur 
in actual performance, Clinical training includes 
observation, bed-side assistance and hands-on-train- 
ing under mentorship (including Tele-mentoring) 
and proctoring [4,7,10]. 

Simulators can be classified as low fidelity, high 
fidelity, virtual reality and augmented reality [4— 
7,11,12"], Low fidelity simulators, like Dry lab lap- 
aroscopic box trainer, are portable, less expensive 
and have been proven to improve surgical skills over 
time. But, they have disadvantages of lack of dupli- 
cation of a real surgical environment, lack of feed- 
back and inability to teach an entire procedure. 
High fidelity simulators include animal models, 
cadavers and commercially available models. They 
have advantages of providing a more realistic 
environment for training, but also have disadvan- 
tages such as lack of easy availability, cost, ethical 
issues, veterinary assistance, anatomical variance 
from human organs (with animal models) and lack 
of bleeding and actual tissue compliance (for cadav- 
ers). The Virtual Reality simulator utilizes a com- 
puter-derived realistic virtual operative field with 
tactile feedback on laparoscopic instruments. The 
Augmented Reality simulator provides a more real- 
istic procedure-specific operating environment, 
where events on the field are enhanced and supple- 
mented [12,1314]. 

Simulators enable residents and novice robotic 
surgeons to practice their skills in a nonclinical 
environment, any number of times, without risking 
the actual patients. Moreover, they provide trainees 
a platform to assess their performance and keep 
track of progress over time. Additionally, they pro- 
vide an opportunity to a surgeon to refamiliarize 
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himself with the surgical console immediately 
before a case as a ‘warm-up’ before surgery [4-10]. 

‘The simulator training can be further classified 
into two types ~ skills training and procedure-based 
training [4-6]. Most of the virtual reality simulators 
provide skills training including cutting, depth per- 
ception, hand-eye coordination, suturing and 
retraction, Recently, procedure-based training simu- 
lators have been reported, which can act as a bridge 
between formal and informal training [13,14,15"].. 

In this systematic review, we have reviewed all 
publications in PubMed in the last 12 months using 
keywords: simulation, robotic training, virtual real- 
ity, augmented reality. We will discuss the current 
status of all existing simulators in robotic training 
including their advantages, disadvantages, all 
recently published modifications in simulators tech- 
nology, assessing their place in current robotic train- 
ing curriculum, along with the recent developments 
in simulator technology and future challenges in 
the simulator training for acquisition of robotic 
skills, 


VALIDATION OF SIMULATORS 

Although simulators have shown their utility over 
other educational tools like didactic teaching and 
dry lab training, they need to be validated before 
their effective integration into teaching and train- 
ing curriculum [4-6]. Validation can be subjective 
and objective. The subjective validation includes 
face and content validity. Face validity is defined 
as the informal assessment of realism and feel by no 
experts. Content validity is defined as the formal 
assessment of appropriateness as a teaching tool by 
experts. The objective validation, which is a much 
‘more daunting task, includes construct, concurrent 
and predictive validity. Construct validity is defined 
as the ability of a simulator to discriminate experts 
from novices. The term ‘novice’ includes subjects 
with no experience at all in performing the pro- 
cedure under study. The term ‘expert’ includes sub- 
jects with adequate experience in performing the 
procedure under study. Concurrent validity is 
defined as the ability to compare performance on 
a simulator with gold standard tests known to 
‘measure the same domain, such as tissue or animal 
lab. Predictive validity is defined as the ability to 
predict future performance based on performance 
on the simulator [4~10]. 


VIRTUAL REALITY SIMULATORS 


We found five different types of virtual reality simu- 
lators published so far in the literature. 
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SimSurgery Educational Platform Robot 

‘The SimSurgery Educational Platform (SEP) Robot 
(SimSurgery, Oslo, Norway) is a modification of the 
SEP Basic laparoscopic virtual reality simulator. It 
replaces the simulated laparoscopic instruments 
with the wristed instruments found in the da Vinci 
robot, providing seven degrees of freedom. It does 
not provide three-dimensional images, fourth arm 
integration or performance feedback. It also does 
not include the following tasks: camera and clutch- 
ing; needle control and driving; energy and dissec- 
tion [9,16]. The experience with this simulator is not 
as robust as with other simulators, though it is an 
extremely cost-effective alternative. However, the 
face, content and construct validity have been pro- 
ven in literature [9,16]. 


Robotic Surgical Simulator 

The Robotic Surgical Simulator (RoSS) is another 
type of virtual reality simulator offering 16 modules 
with progressive difficulty from pinching, camera 
and clutch operation to tissue cutting and cautery. It 
is a stand-alone system mimicking da Vinci Surgical 
System, It helps in developing motor and cognitive 
skills for performing robotic surgery by providing in- 
vivo virtual operative steps with three levels of 
complexity in the form of modules for orientation, 
motor skills, basic surgical skills and intermediate 
surgical skills [17]. The face and content validity 
have been published for this simulator, but there 
is currently no literature on construct validity [4,9]. 
‘The educational impact of this simulator has been 
published as those trained on RoSS took less time to 
complete robotic dry tasks [18] 


ProMIs 
‘The ProMIS hybrid simulator (Canadian Aviation 
Electronics Healthcare, Canada) has a computer 
and a laparoscopic interface made with a plastic 
mannequin with a black Neoprene cover. There are 
three camera tracking systems to detect any instru- 
ment inside the simulator from three angles, thus 
recording the three-dimensional position of tips of 
instruments 30 times/second. It can be used for var- 
{ous tasks like intracorporeal suturing, precision cut- 
ting, cannulation and peg transfer, analyzing three 
objective parameters of time, path and smoothness 
[19]. The face, content and construct validity have 
been reported in published literature [9,19]. 


Mimic dV-Trainer 


dV-Trainer (4V1) is a table top-sized compact system 
with dual-platform capability simulating both 
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da Vinci S, Si and Xi robots. It utilizes precise 
‘modelling of robot kinematics, foot pedals and mas- 
ter grips. This provides trainees with a realistic 
representation of the da Vinci system. This provides 
both basic (Endowrist’ manipulation, camera, 
clutching, and troubleshooting) and advanced skills 
training (needle control and driving, suture and 
knot tying, energy and dissection) [4,7]. The face, 
content, construct validity and educational impact 
have been proven in recent published series 
[6,18,20-22]. Schreuder et al. evaluated 42 partici- 
ants in three groups according to their robotic 
experience. Experts performed better in terms of 
‘time to complete’ and ‘economy of motion’ in 
comparison to novices [20]. 


da Vinci Skills Simulator 


‘This simulator, produced by Intuitive Surgical, can 
bbe integrated with existing da Vinci Xi or Si surgeon 
consoles, thus providing a practice platform to be 
used inside or outside the operating room, with no 
requirement of additional system components. This 
was developed in collaboration with Mimic Tech- 
nologies and Simbionix and provides training 
modules from basic to advanced skills including 
Endowrist manipulation, camera and clutching, 
fourth arm integration, needle control and driving, 
energy and dissection [4,23]. The face, content and 
construct validity have been proven in the recent 
series [11,18,24~28). Tergas et al. showed that train- 
ing on da Vinci Skills Simulator (dVSS) resulted in 
significant improvement in ‘time to completion’ 
and ‘economy of motion’ for novices [24]. They 
found that autonomy of use, computerized perform- 
ance feedback and ease of setup were unique advan- 
tages to dVSS, thus providing more efficient and 
sophisticated training in comparison to conven- 
tional dry laboratory training. 


AUGMENTED REALITY SIMULATORS 

‘These simulators provide a more realistic operating 
field to trainees, utilizing enhanced and supple- 
‘mented events [29]. 


Hands-on-Surgical Training 
‘This simulator is a mode embedded within the RoSS 
simulator and provides training in actual surgical 
cases such as radical prostatectomy, radical cystec- 
tomy, radical hysterectomy and extended lymph 
node dissection. It includes integrated user inter- 
action, narrative instructions and guided move- 
ments. Hands-on-Surgical ‘Training (HoST) was 
created by augmenting a real surgical procedure 
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within a virtual reality framework utilizing audio- 
visual explanations and anatomically relevant illus- 
trations of the critical steps of the procedure. The 
RoSS manipulators navigate the trainee through 
haptic-enabled cues during the procedure [13}. 
Chowriappa et al. [12] evaluated the role of aug- 
mented reality-based skills training for robot- 
assisted urethrovesical anastomosis in a randomized 
controlled trial, using HoST a technology group and 
control group. They found that for 70% of partici- 
pants, HoST the training experience was similar toa 
real surgical procedure and 75% of trainees 
responded that this training could improve confi- 
dence in performing a real procedure. They con- 
cluded that training with HoST in urethrovesical 
anastomosis improves technical skills acquisition 
with minimal cognitive demand. 


Maestro AR 
‘This was introduced by Mimic Technology, provid- 
ing virtual instruments for interaction with 
anatomy in a 3D video environment. This has been 
designed for training novices in decision-making 
skills and procedure-specific skills, within the VT 
simulator. The participants use virtual robotic 
instruments in anatomical regions collected from 
3D surgical video. This simulator plans to provide 
training in four modules: partial nephrectomy 
(released May 2014), hysterectomy, prostatectomy 
and general surgery (to be released) by helping to 
identify anatomy, anticipate tissue retractions and 
predict regions for dissection [14]. There are no 
studies documenting face, content, construct, con- 
current and predictive validity of this simulator, 
‘owing to its recent introduction, 


RECENT DEVELOPMENTS IN CONCEPTS 
Recently, more simulation models have been 
launched emphasizing the concept of teamwork 
and procedure-specific training in robotics. 


Xperience Team Trainer 
‘This simulator, available as an optional hardware 
complement for the dV~Trainer simulator, has been 
introduced to emphasize the importance of team- 
‘work and proper coordination between console sur- 
geon and assistant during robotic surgery. This 
simulator provides training simultaneously to both 
surgeon and bedside assistant. Thus, the bedside 
assistant performs basic skills exercises, promoting 
his psychomotor skills and rehearsal of interaction 
with console surgeon. It also exposes them to real- 
life situations in the operating room, promoting 
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patient safety. Moreover, this team training helps 
in development of communication protocol in the 
real operating room using a well tolerated simu- 
lation environment. Moreover, it also provides pro- 
ficiency-based scoring for the team and each 
individual [30]. However, studies regarding its face, 
content, construct, concurrent and_ predictive 
validity are still pending because of its recent intro- 
duction. 


‘Tube 3 module with dV-Trainer 
This simulator training emphasizes procedure- 
specific training, utilizing the Tube 3 module in 
the VT. It helps in increasing vesicourethral anas- 
tomosis (VUA) performance, one of the most com- 
plex steps in robot-assisted radical prostatectomy. 
Kang etal. [15™] recently published their experience 
with this module. They found that experts. per- 
formed better in task time, total score, total 
economy of motion and number of instrument 
collisions in comparison with novices. Moreover, 
80% of experts found this module a useful training 
tool to perform VUA. Thus, they reported face, 
content and construct validity of the Tube 3 module 
for practicing VUA. 


RobotiX Mentor 


‘This simulator has been introduced recently provid- 
ing a realistic representation of the work space, 
master controllers, pedals and surgeon console of 
da Vinci Surgical System. It provides a 3D high- 
definition stereoscopic view for basic skills (robotic 
suturing, stapler, Fundamentals of Robotic Surgery 
modules) and multidisciplinary complete virtual 
reality procedures (vaginal cuff closure, hyster- 
ectomy modules), augmented with step-by-step 
video guidance and realistic representation of emer- 
gency situations and complications. The trainees are 
provided with performance reports with learning 
curve graphs utilizing simulator curricula manage- 
ment system [31]. However, face, content, con- 
struct, concurrent, and predictive validity of this 
simulator have not been proved in literature because 
of its recent introduction. 

‘Table 1 shows comparison between the available 
simulators. 


CURRENT CHALLENGES AND FUTURE 
PERSPECTIVES 

‘The definitions of face, content, construct, concur 
rent and predictive validity need to be standardized 
for all simulators and future studies. Very few 
randomized controlled trials (RCTs) have been 
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reported comparing different robotic simulators 
[32]. The superiority of one simulator over another 
has not been established so far because of a lack of 
these RCTs. There are no studies documenting the 
actual benefits of simulator training carried over to 
real-case performance with a surgical robot. The 


cost of these simulators is a significant matter of 
concern [4,7-9]. However, with increasing use of 
robotic technology and increasing competition 
among training devices, the future cost of these 
devices should come down to an affordable range. 
‘There is a need to provide more procedure-specific 
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‘elomentoring teleproctoring) 
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Virtual reality simulator 
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FIGURE 1. Potential role of simulators in robotics training, 
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‘training along with skills-based training in a more 
realistic augmented reality environment like HoST 
and Maestro [13,14]. Moreover, the concepts of 
teamworking, decision-making and = communi- 
cation skills should be incorporated more in simu- 
lator training by providing team-based robotic 
simulation environments like Xperience Team 
‘Trainer [4,7-9,30]. However, their validations have 
to be proved in future large prospective RCTs. 
Finally, there is a need for standardization for train- 
ing and credentialing in robotic surgery as has been 
done with Fundamentals of Laparoscopy Surgery for 
laparoscopy in general surgery [4,7,8]. A similar 
standard and validated tool including simulator 
‘training and other training tools needs to be incorp- 
rated in various robotic residency and fellowship 
teaching curriculum (Fig. 1), 

‘There are a few limitations of this article. First, 
we may have missed a few articles related to the 
current topic. Second, we could not discuss certain 
issues like cost-effectiveness, concurrent and predic- 
tive validity (tools to assess the actual benefits of 
simulator training carried over during real-time 
robotic surgery), as these issues have not been 
reported in published series. 





CONCLUSION 

‘The simulator training can form an integral part of 
credentialing and training robotic surgery of future 
robotic surgeons. It has the potential to decrease 
the learning curve for the acquisition of robotic 
skills. It can supplement the hands-on training 
clinical phase and can act as a bridge between 
preclinical training (didactic lectures, dry lab train- 
ing, animal models) and actual hands-on clinical 
training without jeopardising the safety of 
patients. There is a need for more procedure- 
specific augmented reality simulator training in a 
cost-effective manner, utilizing advancements in 
computing and graphical capabilities for new 
innovations in simulator technology, with empha- 
sis on both technical skills training and teamwork 
training. However, more RCTs involving larger 
numbers of participants are required to establish 
its cost-benefit ratio along with concurrent and 
predictive validity. 
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ABSTRACT: 


Background: Objective quantification of surgical skill is imperative as we enter a 
healthcare environment of quality improvement and performance-based reimbursement. 

The gold standard tools are infrequently used due to time-intensiveness, cost-inefficiency, 
and lack of standard practices. We hypothesized that valid performance scores of surgical 


skill can be obtained through crowd-sourcing. 


‘Methods: Twelve surgeons of varying robotic surgical experience performed live porcine 
robotic-assisted urinary bladder closures. Blinded video-recorded performances were 
scored by expert surgeon graders and by Amazon.com Mechanical Turk crowd-soureing 
crowd workers using the Global Evaluative Assessment of Robotic Surgery (GEARS) 


tool asses 





ing five technical skills domains. Seven expert graders and 50 unique 


‘sment 





Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global ass 


scores were analyzed for correlation and agreement. 


Results: Six-hundred Mechanical Turkers completed the surveys in under 5 hours, while 
7 surgeon graders took 14 days. The duration of video clips ranged from 2-11 minutes. 


The correlation coefficient between the Turkers and expert graders’ scores was 0.95 and 





Cronbach's alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89. 





Conclusion: Crowd-sourcing surgical skills assessment yielded rapid, inexpensive 
agreement with global performance scores given by expert surgeon graders. The crowd 
sourcing method may provide surgical educators and medical institutions with a 
boundless number of procedural skills assessors to efficiently quantify technical skills for 


use in trainee advancement and hospital quality improvement. 


INTRODUCTION: 

The healthcare environment is shifting towards performance-based 
reimbursement and focusing on quality improvement. A 2000 study from the Agency for 
Healtheare Research and Quality showed that surgical mortality is among the top 10 
causes of death in the United States. [REF] While not all deaths from surgery were due to 
technical error in this particular report, another study evaluating the role of surgical 
traince’s in malpractice claims data showed that errors in manual technique were present 
in 56% of all cases.[REF] 

Recent literature has shown that blinded video assessments of technical 
performances among experienced laparoscopic surgeons directly correlates with patient 
‘outcomes.[REF] Subsequently, efforts have been made to adopt methods for evaluating 
technical skill using global surgical performance-rating scales, like the Objective 
Structured Assessment of Technical Skills (OSATS) tool, and its derivatives like GEARS 
or GOALS tools.[REF] These methods are validated and are considered the gold-standard 
for evaluating surgical technical skill, but they are resource and time intensive, require 
the time of busy surgeons to evaluate videos, and are thus infrequently used. A cheaper, 
faster, less-biased means of assessing technical skill is needed. 

Crowd-sourcing is a means of accomplishing tasks through the work of 
decentralized, independent groups of people who are generally non-experts in the task 
that is being accomplished.[REF] The advent of the internet has enabled a global labor 
market ready to do various tasks/surveys to help solve problems, from helping blind 
mobile phone users navigate their surroundings,[REF] to discovering complex protein 
folding structures [REF] In recent studies, crowds have been shown to be as effective as 
expert surgeons at evaluating surgical technical skill in a dry lab setting, [Chen et al. and 
Holst et al.] Not only did the crowds perform as effectively as the expert surgeons in 
providing skill assessment, but the costs, efficiency, and practicality of use were all 


improved with crowd graders vs. expert surgeon graders. The major limitation of these 





studies was that the surgical tasks being assessed were dry-lab tasks. Thus, no real tissue 


‘was being manipulated in the study leaving questions regarding whether non-experts can 





appreciate the subtlety of real surgery. In this study, we hypothesize that crowd-sourcing 


can be used to obtain valid performance grading of surgical technical skill on real, living, 


viable tissue. 


MATERIALS AND METHODS: 

After institutional review board approval, two reviewer groups were recruited for 
this study: Amazon.com Mechanical Turk™(Amazon.com, Seattle, WA) users and expert 
faculty surgeon graders, who have expertise in robotic surgery. Surgeons were recruited 
via email and crowd-workers were recruited through the Amazon.com Mechanical 
Turk™ platform, Six-hundred pre-qualified Mechanical Turkers™ were recruited for the 
study (Figure 1). In order to qualify for the study, the crowd-workers had to have 
previously completed 100 or more Human Intelligence Tasks (HITs), the task unit used 
by Mechanical Turk™, and must have had a greater than 95% approval rating as 
qualified by the Mechanical Turk™ as described in Chen et al.{REF] These workers were 


identified only by a unique, anonymous user identific 





mn code provided by Mechanical 


Turk™ and no other information was known about them (gender, age, sex, ethnici 





etc.), Each crowd-worker was compensated $0.75 USD for participating. The expert 
faculty surgeon grader group consisted of seven experienced robotic surgeons, who have 
all practiced as attending surgeons for a minimum of three years with predominantly 
minimally invasive surgery practices and who were familiar with evaluating surgical 
performances by video analysis. The expert surgeons did not receive monetary 
compensation. All graders were required to be over 18 years old. 

A surgical skill assessment survey was developed and hosted online on a secure 
server. The survey consisted of an initial qualification question in which the crowd 
reviewers were shown two videos of a pair of surgeons performing a Robotic 
Fundamentals of Laparoscopic Surgery (RFLS) block transfer task displayed side-by-side 
(Figure 2). The surgeon in the left video performed the task with a high level of skill 
while the surgeon in the right video performed the task with intermediate level of skill. 


The skill level was based on published benchmark metries for this particular task.[REF] 





Crowd-workers were asked to pick the video with the higher level of skill. Crowd- 
workers who incorrectly answered the qualification question were excluded from the 


analysis but were still remunerated. Additionally, the survey also contained an attention 


question to ensure the assessors were actively paying attention and those crowd-workers 
who incorrectly answered were also excluded from the analysis. (FIGURE 1) 
For the second part of the survey, we obtained twelve recorded videos of twelve 


different surgeons of varying skill level performing live porcine robotic-assisted urinary 





bladder closures. (Figure 3). No identifying information of the surgeons performing the 
bladder closures was present and all graders (crowd and expert) were blinded to the 
identity of the surgeons. The length of the videos ranged between two minutes and eleven 
minutes and the average length was four and a half minutes. The videos were uploaded 
to the online survey which incorporated five domains from the Global Evaluative 
Assessment of Robotic Skills (GEARS) validated robotic surgery rating tool — bimanual 
dexterity, depth perception, efficiency, force sensitivity, robotic control.[REF] (FIGURE 
4) Fifty unique Mechanical Turk™ crowd-workers and seven expert surgeons graded 
each video based on the five skills domains. We chose fifty crowd-workers per video 
based on a previous internal analysis of data collected by our team (Chen et al 2014) 
which found 30-50 crowd responses sufficient to achieve satisfactory agreement with 
expert grades.[REF]. 

‘Composite performance scores were tallied by summing the Likert ratings across 
the five domains with a scale of 5-25. The mean of the crowd-worker’s composite scores 
‘was compared to that of the expert faculty surgeons’ and assessed for correlation using 
Cronbach's Alpha statistic to assess the degree of concordance. (Table 1) According to 


‘common practice, levels above 0.9 indicate “excellent agreement,” down to 0.7 “good 





agreement” and levels below 0.5 indicate “poor, unacceptable” levels of 


agreement. [REF] 


RESULTS: 
Afier excluding crowd-workers who failed the attention or discrimination 

question, we were left with valid grades from 487 Mechanical Turk™ crowd-workers. 

(FIGURE 1) It took 4 hours 28 minutes to receive all crowd-worker grades for the 12 


videos. In comparison, it took 14 days to receive grades from all 7 expert surgeons, 





Composite scores given by both the crowds and experts are shown in Table 1. Inter-rater 


reliability between the surgeons and crowd was 0.93 using Cronbach’s Alpha statistic, 


which indicates “excellent” agreement, (Table 1) The linear relationship between the 
surgeon grades and crowd grades is shown in Figure 3. The R? value is 0.91 
DISCUSSION: 


The current gold standard - OSATS-like methods for objectively assessing surgical 





skill - continues to be underutilized due to cost, resource-intensiveness, and the lag-time 
for return of results, [REF] Feedback is most effective if given immediately or near real- 
time so existing OSATS practices tend to be deficient outside of an academic research 
project. [REF DARZI PAPER] Because expert assessment may have significant 
variability in the absence of an ‘agreement workshop” and because mentor bandwidth 
precludes frequent iterative trainee objective technical skills assessment, alternative 
methods to assist in these goals are required. Additionally, OSATS may not be that 
“objective” as the reviews tend to be done by reviewers who are within the same 


institution as the reviewees.[REF]. 


In Holst et al. and Chen et al., it was noted that C-SATS was not designed to 
replace one-on-one instruction and evaluation in the setting of residency training, but 
may provide an adjunct. Traditional methods of instruction and feedback are invaluable 
because they offer content expertise and transfer information about the nuances of 
surgery that could not be yielded by crowds.[REF] C-SATS, however, may have a role in 
rapidly triaging trainees with deficiencies and then allowing mentors to target valuable 
training resources to these deficiencies as opposed to teaching all trainees with the same 
curricula, Feedback from crowds may be obtained rapidly enough to provide guidance 
between surgical cases or between days in the operating room. 

C-SATS has been used in a residency training environment which is ideally suited 
to this method because of the controlled, learner-centered nature of residency. In Holst et 
al,, they showed that crowds can identify differences in Urology resident training levels 
and that crowd-sourcing is a practical, effective way of providing feedback to Urology 
residents in near real-time. [REF Holst J Endourology paper] The major limitation of that 
study, however, was that all tasks evaluated were dry-lab tasks. In a setting of resident 
‘work-hour restrictions, surgical trainees are spending more time in simulation labs to 


refine their technical skills, and thus it is important that crowds can evaluate these dry lab 


tasks quickly. However itis vital to prove that crowds can judge technical skill being 
performed on real, live tissue as opposed to dry-lab materials. Animate surgery better 
approximates real human surgery, thus our hypothesis needed to be tested in this 
environment as a ‘next step’ in validating C-SATS. With no knowledge of relevant 
anatomy, crowds provided extremely rapid and accurate feedback in comparison to 
expert graders. 

A limitation of this study is that only one type of live-tissue performance was 
assessed and the surgery was still a very controlled environment being a porcine lab. In 
addition, all videos assessed were relatively short (averaging under 5 minutes in length). 
It remains to be seen if crowd evaluators can continue to provide effective grading across 
a range of live-tissue surgeries with varying lengths. Future studies aim to include videos 
across a range of surgical approaches such as laparoscopic and open surgeries. Additional 
validation is needed before C-SATS is imbedded into training centers, and evidence that 
crowds can evaluate live-tissue surgery adds to the growing body of evidence for the 


value of this adjunctive objective assessment tool. 





Another limitation to this study is that the performances assessed were from a 
wide range of surgical skill levels from robotic faculty to novice trainees. Thus, the skill 


‘effect 





ize’ may have been disparate enough for lay people to easily see differences. It is 
arguable that if the cohort of performers were of more similar skill levels, it would 

require expert observers to discriminate the smaller technical skills differences. Resident 
training environments where the skills of the trainees vary significantly are ideally suited 


to using this methodology. Additional studies will be needed to test C-SATS on cohorts 





of surgeons who have similar skills 
CONCLUSIONS: We demonstrate that crowd-sourcing basic surgical skills of animate 
surgery compares favorably to a panel of expert surgeon assessors and is faster than the 
experts - providing large-volume feedback in a matter of hours. Utilizing crowd-sourcing 


as a means to assess technical surgical skills provides an inexpensive, scalable, rapid and 





effective way to evaluate live-tissue procedures, paving the way for further v: 


human surgery. Ultimately, C-SATS assessments will need to be linked to clinical 


‘outcomes to gain confidence that presumably non-medically-trained crowds of people 


can accurately ascribe surgical skill. 
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ABSTRACT 


The daVinci Surgical System offers surgeons improved capabilities for performing complex minimally invasive 
procedures; however, there is no standardized assessment of robotic surgeons and a need exists to ensure that a 
‘minimal standard of care is provided to all patients. The Department of Defense and governing surgical societies 
convened consensus conferences to develop a national initiative, resulting in a curriculum called the 
Fundamentals of Robotic Surgery (FRS). FRS is comprised of an online curriculum and a psychomotor skills 
dome. 











This paper describes the production process used to create a psychomotor skills assessment device - the FRS 
Dome. The device was designed to measure the essential skills that are required of any robotic surgeon and to 
provide a basis upon which to grant or deny privileging with the robot. It was constructed to test seven tasks of 
‘manual dexterity: Docking, Ring Tower Transfer, Knot Tying, Suturing, 4" Arm Cutting, Puzzle Piece 
Dissection, and Energy Dissection 








The initial design of the device was created by a committee of experienced minimally invasive surgeons, with a 
background in testing protocols and materials. The design was rendered in computer animation, which kick 
started a prototyping effort with physical materials. These included platinum cure silicone approximating human 
tissue and a 3D polyjet printer for the structural framework. Usability testing was conducted and iterative 
‘modifications were made to improve ergonomics, standardization, and cost requirements. Final CAD diagrams 
‘and specifications were created and distributed to medical and simulation companies for both physical and 
digital manufacturing. This development process demonstrates the evolution of a simulation and a physical 
testing device based on international expert consensus. The specifications are open source, allowing competitive 
production and future iterations. The goal of this paper is to discuss how this device evolved from an idea to a 
‘manufactured product and a digital simulation, 
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INTRODUCTION AND BACKGROUND 


Robotic surgery has been established as an innovative approach in surgery due to a telemanipulator device, 
which introduced a new dimension into surgical tools, This device allows surgeons to manipulate robotic arms 
from a remote console to perform complex surgical procedures. Robotic surgical systems overcome laparoscopic 
limitations and facilitate the performance of minimally invasive surgery due to 3D vision, 7-degree-of-freedom 
instruments, tremor abolition, motion amplification, and stabilization of the camera (Patel et al.. 2013; Hubens, 
Coveliers, Balliu, Ruppert, & Vaneetdeweg, 2003: Blavier, Gaudissart, Cadiére, & Nyssen, 2007). The system 
also offers 10x magnification, wristed instruments, and a third working arm. Currently, the only system is 
Intuitive’s da Vinci Surgical System (Figure 1). 








Figure 1. da Vinci Surgical System 


Robotic surgery has demonstrated safety and effectiveness for urologic, gynecologic, ENT and complex general 
surgery procedures (Barbash, Friedman, Glied, & Steiner, 2014; Serati et al. 2014; Maan, Gibbins, AL-Jabri, & 
D’Souza, 2012; Luca et al. 2013; Zureikat et al., 2013). Exponential growth of minimally invasive procedures, 
particularly robotic-assisted procedures, raises the question of how to assess robotic surgical skills. This device 
also introduces a specific need for training and certification to ensure a minimal standard of care for all patients. 
Some institutions have attempted to develop and validate robotic taining in regards to specific specialties 
(Chitwood et al., 2001; Geller, Schuler, & Boggess, 2011; Grover, Tan, Srivastava, Leung, & Tewari, 2010; 
Chowriappa et al., 2014; Jare & Curet, 2014); however, the lack of a national standard has pushed surgical 
societies (e.g. the Society of American Gastrointestinal and Endoscopic Surgeons and Society of Robotic 
Surgery) to develop a unified approach and standard for robotic skills raining (Zor et al., 2009), 








To develop a comprehensive model for robotic surgery, the Department of Defense, Veterans Administration, 
and fourteen surgical specialty societies convened multiple consensus conferences to create the Fundamentals of 
Robotic Surgery (FRS) curriculum. A similar education and training initiative was implemented for use in 
laparoscopic surgery. which resulted in the Fundamentals of Laparoscopic Surgery (FLS). FRS Conference 
participants included more than 80 subject matter experts (SMEs), consisting of surgeons, psychologists, 
engineers, simulation experts, and medical educators (Smith, Patel, Chauhan, & Satava, 2013). 


‘The committee's vision of FRS was driven by two main goals: to ensure a perfect understanding of the basies of 
robotic surgery and to develop a psychomotor skills program that focused on basic robotic tasks. The intended 
users for this program are novice robotic surgeons, who could be residents or fellows and attending surgeons 
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who have never used the robotic system. The committee began by outlining outcomes measures and metrics, 
which touched on the essential cognitive, psychomotor, and team taining skills. This resulted in a prioritized 
‘matrix of 25 robotic surgery concepts, which is the core material used in the design and development of the FRS 
Curriculum (Smith, Patel, Satava R, 2013). Two assessment tools were created: an online curriculum for 
knowledge and team training skills and a device for psychomotor skill training and evaluation ( Levy, n.d.) 


‘This paper discusses the process for designing and creating the physical device, known as the FRS dome. The 
purpose is to share the evolution of an idea to a usable device. The dome was conceived by experts who 
identified a clear need for robotic education and collectively developed a solution to fill the gap. The medical 
field is a constant progression of new concepts, devices, and technology. This paper also outlines the framework 
for which others can develop and introduce new concepts in medicine and other domains. 


BRAINSTORMING AND CONCEPT DEVELOPMENT 
Exercise Development 


Of the 25 FRS concepts, 16 are directly linked with psychomotor skills. The FRS committee members then 
identified seven exercises that incorporated all 16 skills. These exercices include docking and instrument 
insertion, tower transfer, knot tying, railroad track, 4th arm cutting, puzzle piece dissection, and vessel energy 
dissection (Table 1). Docking and instrument insertion is an essential and unique robotic skill to begin a 
procedure. Failure at this stage of the procedure can compromise the surgery. Ring Tower transfer is a non 
surgical exercise that introduces the utilization of endowrist manipulation and the 7 degrees of freedom to 
surgeons, Knor tying and railroad track are the base of a suturing exercise. The technology introduced in the 
‘risted instruments facilitates the performance of these tasks. 4th arm cutting is another task specific to robotics, 
‘hich tests surgeon's autonomy. The 4th arm allows surgeons to manage three instruments by using a foot pedal 
to switch between working arms. Puccle piece and vessel energy dissection ate critical tasks, which incorporate 
complex articulation of instruments and application of energy (i.e. cauterization and cutting). 





‘Table 1: Description of the basic psychomotor skills attached to the seven ERS tasks, 
Exercises ‘Skills 








Task: Docking & Instrument Insertion, = Docking 
Instrument insertion 

| = Eye-hand coordination 

= Operative field of view 








= Eye-hand coordination 
= Camera navigation 

= Clutching 

= Wrist articulation 

= A-traumatic handling 








Task 3: Knot Tying: = Knot tying 
= Suture handling 
= Eye-hand coordination 


= Wrist articulation 








Task 4: Railvoad Track: = Needle handling & manipulation 
= Wrist articulation 

= A-traumatic handling 

- Eye-hand coordination 
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Task 





4 Arm Cuttin ~ Multiple arm control & switch 
= Cuting 

~ Actraumatic handling 

= Eye-hand coordination 





Task 6: Puzzle Piece Dissection: ~ Sharp and blunt dissection 
= Cutting 

= Actraumatic handling 

- Eye-hand coordination 

= Wrist articulation 








Task 





7: Vessel Energy Dissection! = Energy sources use 
= Sharp dissection 

= Cutting 

= Multiple arm control 

= A-traumatic handling 

= Eye-hand coordination 








Device Development 


‘The FRS committee envisioned all of the exercises contained on the outer surface of a single device. This would 
allow for the exercises to be administered quickly and easily. incur less cost, and ensure uncomplicated storage 
and transportation. The semi-spherical form (ie. the dome), was quickly decided on as a shape which would 
integrate with the current robotic system. They depicted their ideas through simple drawings and crude models 
‘made from materials found on hand. During initial design planning, conference participants experimented with 
variety of arrangements of the exercises on the dome. 





‘A final sketch was developed and delivered to a 3D digital artist to create static pictures of the device, along with 
an animation of the performance of each exercise. The CGI provided the first formal images of the dome, which 
‘gave life tothe device and proved feasibility. The realistic animations showed the exercises being performed and 
gave committee members a visual concept of how the device would function (Figure 2). 





PROTOTYPING 


‘The prototyping process began using the ideas developed in the design meeting and the CGI. This process would 
prove to be fundamental in confirming the design expectations. It was essential to determine if a single device 
could physically house all of the exercises effectively. if the planned architecture was compatible with the 
robotic system, and if the outcomes of the exercises could be measurable and reproducible. 


Low-fidelity Prototypes 
Low-fidelity prototypes (LFPs) were created using simple and inexpensive materials. None of the materials used 
in the LEPs were intended for inclusion in a final product. These materials were chosen because they were 


readily available, inexpensive, and easy t0 manipulate to tes fit and function, These materials allowed rapid trial 
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and error testing of the technical aspects, clarifying requirements, and proving usability. The testing of the LFPs 
‘was performed using the da Vinci Surgical System and was video recorded. These recordings were sent to FRS 
committee members to provide their feedback. Each LEP resulted in multiple improvements to the designs, 
hich were tested on subsequent prototype versions, 


‘The base model of the LFPs was created using half of an 8” Styrofoam sphere as the support structure, yellow 
felt material as the fat layer, a latex swimming cap for the skin layer, and straws for the embedded vessels. The 
base of the towers was constructed using synthetic foam blocks carved into a cone shape (Figure 3). The exercise 
patterns were drawn onto the surface using a permanent marker. 





Figure 3. Base of Low Fidelity Prototypes 


‘The LFPs evolved over six iterations, all of which introduced design improvements (Figure 4). At the earliest 
phase in LEP testing. it was quickly realized that the dome size Was too large 10 fit under the robot arms 
appropriately. So, the dome size was decreased from 8” to 7°, Another modification made early in the LEP 
development was to change the 4th arm cutting band from a rigid tube to an elastic band. This allowed for the 
user to adequately stretch the band prior to each cut, 





Figure 4. Iterations of LEPs 


‘The suturing and dissection exercises involved the most modifications during the LFP stages. The original 
cloverleaf shape, used for the dissection exercise, was found to be too large and did not allow for the surgeons 10 
access the section of the shape that was located on the backside of the dome. The size of the pattern was reduced: 
however, this did not mitigate the accessibility issue. The team experimented with other options, such as splitting 
the clover leaf into three sections and adding smaller shapes to the center of the cuting area. This design was not 
practical because once the smaller shapes Were cut, the latex receded and inhibited surgeons from cutting the 
surrounding shape. 








Eventually, the dissection shape evolved to a puzzle piece that incorporated all of the prerequisites for the 
dissection exercise (i.e. an accessible shape and a complex design). By using this compact pattern it became 
clear that all exercises could be grouped into an area covering only one thied of the surface of the dome. This, 
‘opened the opportunity to replicate the cluster of exercises three times on the surface, reducing the materials and 
costs for repeatedly practicing with the device. Another obstacle was to build the suturing exercise with the 
adequate materials and placements, to ensure a realistic feeling of suturing. Originally, the incision was made 
into the latex swim cap, however the latex would tear away and recede after the incision was cut in this model, 
‘Two versions of the suture module were experimented with: an embedded silicone and an external latex model. 
Eventually the embedded silicone model was chosen as the most realistic and practical for the exercise. 
‘Ultimately, the basic structural changes found in the low-fidelity prototyping were: 

The dome hase needed to be reduced to 7” 

‘The dome base needed to be substantial in weight to keep from moving under the force of the robot 

* Assmaller, yet equally complex dissection shape was necessary 

©The exercise sets could be grouped to allow them to be repeated on the surface of a single dome 
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‘= ‘The magnets which held the towers to the dome needed to be of sufficient strength to hold through the 
layers of fat and skin 


High-Fidelity Prototypes 


‘The high-fidelity prototypes (HFPs) were made using higher quality, custom materials. These materials had the 
desired qualities of the final product and could be used as a basis for the large scale manufacturing process, The 
styrofoam base fom the LFPs was replaced with a support structure that was printed using a 3D polyjet printer 
(Figure 5). A polyjet type 3D printer works similarly to an inkjet printer in that it distributes layers of polymer to 
build the desired design, which is cured by UV light. This type of printer was chosen because of the versatility 
allowed by printing multiple materials at once. Also, the jet lays 161m layers of liquid polymer, which gives 
printed parts a finer resolution. Using this printer, a dome shell with a lid was created. The shell and lid had 
divots covering the surface, allowing for magnets to be moved to many different placements on the dome during 
design experiments. A small jig was also created using the 3D printer. Prior to the creation of the jig, the wires, 
were made by hand, but the jig enables the standardized creation of the S-shaped and -shaped tower wires. The 
price to print these items was approximately $1,000. 




















‘The synthetic tissue layers were created using Smooth-On platinum cure silicone products. These are two part 
silicones, which can be colored and mixed with other additives to achieve the desired product attributes such as 
durometer. The silicone used for the “fat” layer gave a gel-like and slightly sticky texture (Eco-flex Gel), while 
the “skin” silicone had a more firm and non-sticky quality (Ecoflex-04)30). These silicones were chosen because 
they gave the closest resemblance to actual tissue properties, The fat silicone was poured directly onto the dome 
to the desired thickness. A clay mold was then made to replicate that thickness, which was used to form the skin 
layer (Figure 6). Embedded in the skin was a layer of polyester mesh, which helped to provide structure and 
stability of the skin, Small vessels were also created by quickly curing the silicone to a small tube, Using these 
materials we were able to create a set of synthetic tissues for less than $20. 








Figure 6, Pouring of silicones and first HEP 


‘The puzzle piece shape and the other markers were drawn on the skin surface using a permanent marker. The 
exercises were drawn on in different locations, sizes, and orientations for the first HEP. After testing the HEP on 
the robotic system, we finalized the size and orientation of the exercises on this new dome. This is important 
because as learned in the LFP stage, the exercises needed to be placed strategically to compensate for the range 
of movement of the robotic arms. Despite having 7 degree-of-freedom instruments, there are stil limitations 10 
the amplitude of the movement of the robotic arms. We also determined that three trials of each exercise could 
fit on one dome, so each work station (Le. group of exercises) repeated at 120 degree increments on the dome. 
Eventually, we determined that after dissecting the three vessels significant space was available for more 
dissection in the fat layer. So, we added three additional vessels located to the right of the original vessels and 
out of range of potential damage from other exercises (Figure 7). By doing so, the fat could be used six times 
and the skin used three times, which incurs lower costs for the materials used during trai 
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Figure 7. Vessel placement on dome and in fat 


Over many iterative models, we improved our techniques and experimented with different materials and 
additives to achieve the desired qualities. For example we began adding a Thixotropic additive to thicken the 
mixture and allow us to cast the material onto a curved surface. We also tested different inks and techniques of 
printing the shapes and markers on the skin: however, most inks and paints cannot be used on silicone. We 
decided to use a silicone based paint product, which cured the design to the silicone surface. 





We 3D printed miniature dome models (2” in diameter) to begin testing molding materials. We created silicone 
‘molds and used a urethane plastic to cast the model. By doing this we realized that the original 3D printed 
‘material was porous and caused bubbling in the molding, leading to surface bubbles on casted models. So, a new 
full sized dome was printed in a smoother and less porous material, which would be better for manufacturing, 
‘The new dome shell and cap was designed with divots only at the locations necessary for holding a tower 


(Figure 8) 


Figure 8. Final 3D printed dome shell 

















Since this device will be used for training and education, a high level of standardization is necessary. For this we 
added small markers that ensure the pieces are assembled correctly and in a standardized manner for all 
participants. Table 3 details the standardization pieces, 


‘Table 3. Description of the Standardization Markers 





‘Standardization Markers 





Tower tongues sed to orient the towers in the correct direction for each exercise. 








“Triangle in id sed to show proper orientation ofthe towers that are placed inthe ap. 
‘The towers are placed in the two locations directly in line with the 
> | puzzle piece and with the tower tongues on the corresponding line of 
) the triangle. This ensures that the S-shaped towers face the correct 
direction forall users. 











‘Tower orientation markers | These markers aro used to show the placement of the towers on the skin 

and the orientation of the tower. The towers are placed on the marker 
‘with the tongue aligned with the tongue mark. This ensures that all 
towers face the correct way, 
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‘Triangles on dome shell 


“These simall markers are Tocated at 120 degree increments on the Tower 
edge of the dome. They signify where the embedded vessels should be 
located when the tissue layers are placed on the shel 





Triangles on fat 





‘There are two types of triangle markers on the fat: open and closed. The 
closed triangles indicate the location of the first use vessels, When the 
fat is placed on the dome, the closed triangle is aligned with the triangle 
‘marker on the dome shell. After all three vessels are used, the fat is 
rotated and the open triangles are aligned with the triangles on the 
dome. This ensures that the vessels are in the accurate location for the 
dissection exercises. 





‘Triangles on skin 





‘The iiangle markers on the skin are aligned with the irangles on the fat 
layer. These ensure that the puzzle piece lies directly aver the vessel and 
that the tower markers align with the underlying magnets, 








‘ap placement notch 





The noich in the cap ensures that users place the cap Im the comect 
orientation. Since the magnet divots are placed in the shape of a 
triangle, the cap has to be secured in a specific orientation for the 
‘magnet divots to align properly 








In the final HEP, the exercises existed as they would in the manufacturing phase. Final testing was performed in 
order to ensure that all specifications were correct and to build a specifications document, which was used to 
create final CGI and CAD files (Figure 9). 





PRODUCTION 


Figure 9. Final HEP. 


The final CGI, CAD, and specification document were sent to the manufacturing company and simulation 
companies to assist them in their development of physical and virtual domes (Figure 10). 
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Figure 10. Final CGI 


A local manufacturer, familiar with the materials used during prototype testing, used the dome and performed all 
of the exercises prior to beginning the process. This provided a first-hand experience of why certain material 
qualities were so important. The goals for this phase, in addition to mass production, were to maintain device 
integrity and minimize cost. Some of the materials used during prototyping were more expensive than what 
would be feasible for taining centers, For example the $1,000 materials cost for the 3D printed dome was 
reduced to less than $25, 


‘The simulation exercises of the FRS dome will be incorporated into two simulators: the da Vinci Skills 
Simulator (dVSS) and the Mimic dV-Trainer (Figure 11). Both systems contain the six FRS exercises, but vary 
in their software and hardware. The dVSS is a simulation system. which integrates with the actual console of the 
surgical system. This allows users to tain using the exact hardware that they use when operating. The dV- 
“Trainer is a standalone system that uses custom hardware and software. These simulations give the users 
experience performing the FRS exercises without requiring the use of the entire robotic surgical system, 
Generally, the systems are dedicated resources to the hospital surgical department and difficult to reserve for 
training purposes. The simulators also allow unlimited practice sessions without consuming the physical 
materials of the dome. The research team worked with each of the simulator companies to create and test 
‘multiple prototype versions of the exercise software. Our extensive experience with the real materials and our 
surgeons" experience with human surgery allowed us to critically evaluate the simulated behaviors of materials 
and the scoring methods. This feedback has led to significant improvements in the accuracy and usability of the 
simulators 








Figure 11. Mimic dV-Trainer and Simbionix's dVSS simulated dome exercises 


Maintaining the simulated physical properties of the dome was paramount. Since the simulations may be used 
without proctors, the physical behaviors have a considerable impact on the scoring metrics and guidance that is 
given for improving performance. The research team evaluated the simulated exercise properties including 
elasticity of materials, flexibility of sutures, simulated gravity, and the effects of excess force on the virtual 
device to ensure that it behaved similar to the real dome. ‘The real materials however were also limiting to some 
of the desired qualities, particularly in the vessel dissection exercise. The silicon-based materials act as 
insulators, preventing cauterization of the small vessel. Both simulators allow the user to apply energy for 
cauterization, as well as receiving a visual indication that the vessel is losing blood, prompting the user to 
‘manage the situation appropriately. 


Some of the metries also varied between the physical and simulated domes. While the physical dome is scored 
via expert video reviewing, the simulator can more objectively assess a user's performance. This allows the 
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simulated exercises to score some errors more accurately, such as instruments being out of view for a specific 
amount of time and over a specific distance. 


The research team will include these simulations in a pilot study and provide the simulation companies further 
formative feedback on the usability of their systems, to mitigate complications that may occur during the larger 
‘multisite validation study that will follow. This pilot study will also establish preliminary scoring benchmarks 
based on expert performance, which will be used to guide the multi-site validation study. 





CONCLUSION 


Over the course of two years, we created an easily integrated device, using low cost but high-quality materials. 
‘This paper outlines the steps of the FRS dome from idea conception to the development of physical and vietwal 
devices. The goal of this paper is to share the evolution and process for others interested in training and 
assessment devices. Since the FRS dome specifications are open-source, this also serves as an important 
resource for potential producers. 


‘We have taken away several lessons from our experimentation that made our process a success including having 
a multidisciplinary team, soliciting frequent feedback, using easily adaptable designs, testing on small models, 
and using commercial materials during prototyping. Our multidisciplinary team of surgeons and engineers 
allowed for a diverse perspective during the construction of the device. The design changed many times and it 
was beneficial to start off using basic models that accommodated the varying designs. It was advantageous to 
work with actual manufacturing materials once we developed a functional prototype to better envision the final 
product and allow a smoother transition to the manufacturing phase. We recommend testing materials on small 
‘models, which will help cut time and costs. Finally if possible, work closely with the manufacturing teams at an 
early stage of development, particularly when working with virtual models. This will help to flesh out details and 
encourage collaborative development earlier in the process. 


The next step of this work is to conduct formal validation testing of the curriculum including the device and 
related simulations via a pilot and national multi-site validation study. The FRS dome features basic robotic 
surgical skill exercises, which are applicable 1o most specialties. This basic device is scalable and will be the 
foundation for the future, more specialized FRxS devices (eg., the Fundamentals of Robotic Gynecologic 
Surgery (FRGS) and the Fundamentals of Robotic Urologic Surgery (FRUS)). 
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Robotic Simulators: A Case for Return on Investment 
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ABSTRACT 


Simulation has been integrated into the education and certification process in aviation and military arenas with 
significant success in providing cost effective training. The transition from the apprenticeship model to simulation 
has been slower in the field of medicine with cost, lack of curricula and high fidelity exercises and equipment being 
‘the main reasons. With recent improvements in all areas, cost remains a significant challenge. 


‘This report describes our novel analysis of the return on investment (ROI) that can be achieved through the inclusion 
of simulator use within a robotic surgery business practice and as an alternative source of training revenue, 
Information was gathered through an extensive literature review and expert interviews for the development of an 
interactive calculator for institutions to utilize when considering an investment in robotic surgery simulators 


‘This ROI model presents the core improvements to existing operations which may be realized through the use of 
simulators of robotic surgery. Category headings include simulator investment costs, surgeon productivity, surgeon 
health, hospital costs, and other training costs. The user of the model is able to enter their own numbers for their 
unique facilities. The spreadsheet model will calculate the costs and benefits associated with each area, create 
category subtotals, and then an overall total for all areas. Using these numbers, it can then calculate an ROL 
percentage for the simulators. This model represents one tool to assist organizations in making the investment in 
these devices and training programs, 
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Robotic Simulators: A Case for Return on Investment 





Roger D. Smith PhD Khara M. Simpson MD 
Florida Hospital Nicholson Center Columbia University Medical Center 
Celebration, FL 34747 ‘New York, NY 10032 
roger.smith@fhosp.org kmsimpmd@yahoo.com 
INTRODUCTION 


Creating a viable robotic surgery practice within a hospital is an expensive and risky endeavor, The investment in 
equipment, facilities, personnel, and process modification is significant, amounting to millions of dollars in the early 
years of a program. Many hospitals make this investment with a limited understanding of how best to structure a 
robotics practice and the probability of achieving a positive return on this investment. At the business end of a 
decision to create a robotic surgery practice, the hospital has the goals of optimizing the utilization of the robotic 
‘operating room, reducing costs, and ensuring patient safety. Training to and maintaining the competency of the 
surgeons performing the procedures has a direct impact on these areas, allowing the inclusion of simulators in the 
robotic business unit to make valuable contributions. The proposed effects are summarized in Figure 1. This report 
describes our analysis of the return on investment that can be achieved through the inclusion of simulators and their 
regular use within a robotic business practice; and as an alternative source of training revenue. 





‘Surgeon Productivity Training Costs 
Surgeon Stamina OR Costs 
Surgeon Competence Medical Errors 
Surgeon Certification Instrument Breakage 
Surgeon Career Length Insurance Costs 
OR Utilization 





Figure 1, Summary of Simulation Effects on Surgical Practice 


BACKGROUND 


For every complex and expensive system there emerges a need for training devices and scenarios that will assist new 
leamers in mastering the use of the device and understanding how to apply it with value, Intuitive Surgical’s da 
Vinci robot is just such a system. It is currently the only FDA approved device for laparoscopic robotic surgery on 
human patienis. Despite the 1.5 to 2 million dollar price tag, the device has seen rapid distribution and its 
implementation has led to the need to develop more efficient and effective training methods, as well as assessment 
and skill maintenance tools. In laparoscopic surgery, simulators have played an important role in improving the 
practice of surgery over the last 20 years (Schout and Hendriks, 2010; Wohaibi and Bush, 2010). The same trends 
‘and values will likely apply to robotic surgery. 


‘The complexity, criticality, and cost associated with the application of the da Vinei surgical robot have stimulated 
‘the commercial creation of simulators which replicate the operations of this robot. There are currently three different 
simulation systems available for training and developing skills in robotic surgery: da Vinci Skills Simulator 
(Intuitive Surgical Inc.); dV-Trainer (Mimic Technologies, Inc.); and RoSS (Simulated Surgical Skills LLC). Each 
ff these possesses unique traits which make them valuable solutions for different types of users and leaning 
environments. Investment in the simulators alone represents a major capital investment as costs range from $100K 
(RoSS/dV-Trainer) to $600K (da Vinci Skills Simulator) per device and its associated support equipment. Coupled 
‘with increases in direct costs from operating room time and supplies, (Venkat and Chen, 2012; Pasic and Rizzo, 
2010; Bolenz and Gupta, 2010; Bamett and Judd, 2010; Holtz and Miroshnichenko, 2010) robotic surgery 
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implementation can have a profound impact on hospital finances. This doesn’t include costs associated with 
‘ensuring patient safety and surgeon training. From a business perspective, a hospital, college of medicine, or 
robotic practice should identify a return that will be achieved with the purchase of these devices. 


History of Simulation 


Rehearsal and simulation has been one of the primary means of developing and maintaining proficiency in specific 
skills for thousands of years. Lectures and written materials are the primary means for developing cognitive 
knowledge, but these are not effective at instilling psychomotor skills in any field. The skills needed in the hands, 
body, and coordination with the mind must be developed through practice. Rehearsal in a non-lethal environment 
has become the standard of practice for learning in military warfare and aviation. The lives of soldiers and pilots 
today are considered of significant value to demand a structured training program and measure of proficiency before 
centering a life threatening situation. Given the technology that has emerged in the late 20th and early 21st centuries, 
the means of rehearsal have shifted from drills toward events that are mediated by computers with the ability to 
measure performance and provide constructive feedback on means to improve that performance. Implementation of 
direct practice on simulators has led to significant quality improvements and a reduction in training costs to the tune 
of hundreds of millions of dollars. ‘There is a 30 to 40 year history of success documented in the military literature, 
specifically regarding its return on investment. On average simulators cost 5-20% of live training and can reduce 
the length of training by 10-30% (Fletcher and Alexander 2013). Simulation has also been found to be well liked 
by trainees and found to be comparable to live training in terms of experience as well as outcome (Worley and 
Simpson 1996), 











Simulation devices are similarly a prominent tool in medical and surgical education. From the most primitive stuffed 
dolls with anatomical markings used in ancient Chinese societies, to the most current computer driven, three 
dimensional representations of living tissue, simulators have been helping to train surgeons for over a thousand 
years. While there are parallels between the military and medical fields regarding the involvement of human life 
‘and the need for structured high fidelity training, there are some unique challenges. ‘There are many confounders 
in medicine and the seemingly direct relationship between simulation and improved patient outcomes and reduction 
in costs is not robust. So, while most agree that simulation is a necessary requirement for surgical training, the lack 
of evidence coupled with the costs of device and curriculum development have led to a slower adoption of 
simulation in surgical education when compared to industry. One of our goals is to use the lessons from our military 
and aviation industries in conjunction with published literature to identify key variables that impact the ROI of 
robotic surgical simulators and provide a loose framework for how simulators can be better utilized to achieve 
financial, educational, and patient safety benefits. We will focus on the concept of return on investment (ROI) for 
these simulators, not on the capabilities of one specific device. Our goal isto assist the purchasers and users of these 
devices in determining whether such a device is a sound financial investment. 














Pul 





-d Literature 


‘There are very few studies evaluating ROI for the simulators of the robot. There is one article directly evaluating 
the RoSS simulator (Rehman and Raza 2013) where annual training hours were converted to training time on a 
robotic console. In that study, the use of the stand-alone simulator resulted in cost savings of $600,000. Animate 
lab training of the same duration would have cost approximately $72,000 annually to train 100 people each year 
‘The remainder of the studies look solely at laparoscopic simulators, For example, in March 2004, Frost & Sullivan 
Inc. conducted a ROI study on three training simulators sold by Immersion Medical. Specifically, the data regarding 
the Laparoscopy AccuTouch System was published. The Laparoscopy AccuTouch System uses advanced 3D 
technology and graphics to re-create the procedures and environment of abdominal laparoscopic surgery. Using 
reported median values from survey data, financial benefits were estimated at $168,767, based on annual costs, 
‘$76,000, with an estimated payback period of 169 days. Also somewhat related, is a review article by Leddy et al 
that reviewed the published cost analyses of robotic surgery for both urology and pediatric surgery. in an effort to 
develop a novel model for determining return on investment. The premise for the model was that reduction in costs 
directly relate to the ability of the technology to reduce hospital length of stay. Surgical volume can be limited by 
the availability of hospital beds, in which shorter hospital stays lead to greater bed availability, which can lead to 
more procedures being performed. It also acknowledged the learning curve of robotic surgery and the expectation 
for OR times to decrease with improved technology, surgical technique, and time. 
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METHODS 





‘The first step was to identify, at a basic level, the main effects that robotic simulation would have on a healtheare 
organization. We found that the use of a simulator can have a direct impact on at least three major, but separate 
;parts of the healthcare delivery process (Figure 2). 


(Gaal. Outcome 
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Figure 2, Robotic Surgery Simulator Impact Areas 


‘The first, and most important, is patient outcome. The use of a simulator ean potentially improve patient outcome by 
allowing the surgeon to perform more effectively and efficiently with fewer adverse events 


‘The second is the impact on the surgeon. He or she may find that the ability to use the robot effectively impacts their 
career with additional options that can be offered to the patient and to a potential hospital employer. It may also 
reduce the long-term wear and tear on the surgeon's body due to improved ergonomics, making it possible to 
continue practicing surgery fora longer period of time. 


‘Third is the effect that it will have on the costs and efficiency of the business practice. Simulator trained surgeons 
may perform procedures more rapidly, with less error, and lower instances of equipment breakage. This ties directly 
‘back to improved patient outcomes, 


With this as a foundation, we used a diagramming method popularized by Jay Forrester at Massachusetts Institute of 
‘Technology (MIT) to explore the interrelationships between a large number of variables within the complex and 
‘dynamic hospital surgical program. This method identified variables which react almost immediately to changes in 
‘raining offered to robotic surgeons as well as longer-term variables which may take years to emerge as measurable 
retums on the investment. ‘These relationships can be grouped and organized as shown in Figure 3, Note that each 
‘variable is associated with a calculable financial outcome. This model was useful in identifying the factors which 
contributed to ROI in a first, second, and third level of separation. It also motivated discussions around the 
financial and non-financial variables that were changing with the incorporation of simulation, 
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Figure 3. Effects of Simulation-based Training on Robotic Surgery Business 
Expert Opinion 


‘Twelve expert interviews were then conducted at several simulation centers around the country and internationally 
to further expound upon the model, including identifying and quantifying variables. Surgical leaders interviewed 
included Randy Fagin MD, Texas Institute of Robotics; Brendan Sayers PA, University of Texas ~ Southwestern; 
“Amold Advincula MD, Celebration Health; Thomas Lendvay, MD, University of Washington College of Medicine: 
Robert Sweet, MD, University of Minnesota College of Medicine; Col Timothy Brand, MD, Madigan Army 
Medical Center; Jacques Hubert, MD, University of Lorraine Medical School; Dimitrios Stefanidis, MD, Carolinas 
Healthcare System; Martin Martino, MD, Lehigh Valley Health Network; Mona Orady, MD, Cleveland Clinic; John 
Lenihan, MD, MultiCare Health Systems; and Michael Pitter, MD, Newark Beth Israel Medical 








‘There were several common themes identified from the expert interviews. Most agreed that having a defined 
curriculum was imperative to successful training and that it should be identified prior to purchasing a simulator. 
‘These robotic training curriculums consisted mostly of manufacturer-developed didactic education with a 
combination of simulator and/or dry lab assessment. It was also clear that there were a variety of successful 
arrangements for location and proctoring of a robotic simulation curriculum. The majority utilized a combination of 
the dVSS and the dV-Trainer. Only one program had a dedicated training console, therefore most simulation with 
the dVSS was limited to nights and weekends. Simulation was most often a part of resident and fellow curriculums 
but approximately 50% of the sites also had attending level programming. Simulator value was also identified to be 
ted directly to the accessibility of the necessary equipment. 











‘They verbalized a strong need for studies evaluating correlations between simulation and improved clinical 
‘outcomes, determining the ideal length and intervals for simulation training, and comparative studies between 
institutions with and without simulation centers. In addition to maintenance and certification, curricula could also 
bbe used for remediation training following complications, Software interests surrounded the development of 
procedure specific training and increasing complexity of these procedures. Several believed that simulation could 
‘be used globally to decrease insurance costs and increase reimbursements 
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Simulator ROI Model 


Using the information in the system dynamics diagram, we created a basic model of the ROL with short-term, 
‘measurable returns. The model is populated with average numbers from the literature and the authors” institution, 
but the user is able to enter specific numbers representing their unique facilities. Users enter data for their specific 
institution and the model calculates intermediate values and an annual return for Years 1 & 2 following the 
investment, Interested readers may request the full calculator with relevant appendices from the authors at no 
charge, Figure 4 illustrates part of the data entry and calculation fields for the calculator. Table 1 provides a 
sample of the variables and returns that are possible from a simulator-based training program. 





Robotic Simulator Return on Investment Model [ 
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Figure 4. ROI Calculator (Partial Screenshot) 
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‘Table 1. ROI Calculator Sample Variables 
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Listed below are the many factors to be considered in each cost category as well as limitations to the calculator 
Investments 
‘The costs associated with adopting a simulator training program include: 


Cost of Simulator. The initial investment in purchasing the simulator device. The price point for many surgical 
simulators is in the neighborhood of $100,000 per device 


Annual Maintenance Fee. Most devices include one year of maintenance or service warranty with the purchase of 


the device. In following years, there is an annual fee to cover the installation of updates and repairs to the device, 
‘One must also allow for additional repairs beyond the normal maintenance. 
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Facilities Costs. The equipment needs to be housed in a location that is conducive to sufficient access and training. 
‘A dedicated space is preferred, but some institutions use a shared space model to reduce this cost. This impacts the 
‘degree to which surgeons can use the devices on their own initiative which directly impacts training efficiency. 


Staffing Costs. Most simulator devices require a knowledgeable trainer to instruct the surgeons on the use of the 
device and to facilitate data collection and the tracking of performance improvement. This staff person does not 
necessarily have to be dedicated to the robotic simulators, but can be a shared resource that has additional duties in 
the hospital system. The staffing cost can vary widely depending on the type of assignment used, such as hiring new 
personnel versus training an existing employee. An additional consideration would be the loss of revenue from 
physicians, nurses, and other medical professionals during training sessions. 


Supplies & Materials. Because robotic simulators are electronic and use no real consumables like suture pads, 
needles, and fluids, the supplies and materials for continuing to use them is minimal. Some typical supplies include: 
power strips, extension cords, cleaning wipes, spare parts, computer thumb drives, and computer adapters. 


‘The investment in simulator programs is typically highest at initial purchase for the equipment and much lower for 
‘each ensuing year. Therefore, the model separates the first year from years following. The expenses in years 2, 3, 4, 
‘etc. appear to be very similar. At this point there is little data on equipment fatigue with use. There is only published 
data on the RoSS. Master controllers and the pinch devices need to be replaced at 180 and 360 hours of use 
respectively (Rehman 2013). We are therefore not able to identify the effective useful lifetime of a simulator 
device or a significant upturn in costs due to increased maintenance in later years. Insight into what to expect in 
robotics may be found in the operating expenses for laparoscopic training simulators which have been in use for 
many more years. 


Returns 


‘The retums that can be experienced through the implementation of a robotic simulator training program fall into four 
major categories: surgeon productivity, surgeon health, hospital costs, and training costs. Each of these has many 
variables which contribute to financial returns. 


Surgeon Productivity. The primary goal for most organizations creating a simulator training program is to increase 
the productivity of their surgeons. A model of surgeon productivity begins with the number of surgeons who are 
‘rained and the amount by which this training can speed up a typical operation. Reducing the time required to 
perform a surgical procedure can contribute to multiple variables. Firs, it may reduce stafling costs by allowing the 
staff that supports the surgeon to move on to other activities, or to work fewer paid hours during the day. In some 
cases, it may reduce total OR staffing due to the ability of the surgeon to control both the camera and the surgical 
‘arms concurrently. Second and most importantly if the reduction is large enough, it may open a window in the OR 
schedule which is sufficient to perform an additional operation during the day. ‘Third, if simulator-based training 
can reduce the variance in surgical times for procedures, it will make the scheduling of procedures more accurate. 
‘This would reduce the “slippage” that occurs in the daily schedule because a procedure takes twice as long as was 
scheduled. Smaller variances lead to more efficient scheduling which can reduce staffing and facilities costs. This 
model focuses on the impact of reducing surgical time sufficiently to open a window for an additional surgery. The 
impacts of variance across multiple instances of similar procedures is worthy of an entire dedicated study and is not, 
included, 


Surgeon Health, Robotic systems significantly reduce the physical workload experienced by traditional open and 
MIS surgeons. All surgeons have a level of stamina that allows them to perform for a specific number of hours each 
day. Upon reaching this point, the muscles and mental focus of the surgeon are diminished. Continuing to operate 
‘can bring risks both to the health of the patient and the surgeon, 


Rehearsal in a simulator has the effect of training the muscles which are used in surgery. Familiarization with the 
tasks also reduces the amount of mental and physical effort that is required to complete each procedure. The 
extension of surgeon stamina and the reduction in energy expended can both lengthen the number of hours that a 
surgeon is able to perform optimally. 


‘This physical and mental training can extend the surgical day for a specific individual. It may also extend the length 
‘of the surgeon's operating career by reducing repetitive stress injuries and aging due to mental stress. ‘The 
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‘cumulative impact on the length of careers cannot be calculated at this point. Exploration of this effect is left for 
future researchers. 


‘The model of surgeon health included in the ROI model in this paper is limited to daily extensions of operating time. 


Hospital Costs. The model identifies four major hospital cost categories which may be reduced due to improved 
surgeon competence: instrument breakage, liability insurance, surgical error rate, and training costs. 


Inexperienced surgeons can break the controls of the robot through the exertion of too much force and fighting 
‘against the electrical and mechanical components of the robot. When the surgeon damages the robot, the cost of 
many (but not all) repairs is covered by the maintenance agreement; but the significant cost to the hospital is in the 
disruption that occurs in postponing or rescheduling an operation because the robot is no longer operable. 
Inexperienced surgeons can also damage the surgical instruments held by the robot. Each of these typically costs 
between $1,200 and $2,500. Frequent abuses come from forceful instrument collision and friction along the shafts. 
Simulator training can develop skill and dexterity with an instrument which prevents this from happening. 


As medicine becomes even more evidence based, there will be the opportunity to quantify the liabilities associated 
‘with specific surgeon competence levels. This may allow surgeons who can document additional training and 
acquired competence to reduce their medical liability costs when compared with surgeons who have less training. 
Experts whom we interviewed for this study indicated that they had already begun annual mandatory simulation- 
‘based exams as part of their intemal risk management plan with the hope of realizing insurance savings in the future. 





‘The competence that comes from simulator-based training can reduce the number of errors that are made in robotic 
surgery. Errors can require a return to the operating room for a procedure which generates no additional revenue. 


‘There are multiple forms of training used to acquire, maintain, and extend the expertise of the surgeons, When a 
simulator program is introduced it can reduce the need for alternate forms of training. One popular form of robotic 
‘raining today is through visitation and instruction in the OR itself. This long established and traditional practice is 
used in all surgical specialties. But, itis known that “in OR” training results in an extension of the time to complete 
the procedure (Koperna 2004). Additional time is required to explain the procedure and what is happening to the 
trainee. Given the high cost of OR time, a reduction in OR-based live training can generate meaningful savings for a 
hospital 








‘There are two typical forms of outsourced training. The first is bringing an instructor into the hospital system to 
instruct the surgeons. The second is sending the surgeons to other facilities or congress meetings to acquire the skills 
they need. Both of these are subject to reductions when similar skills can be acquired and measured in the simulator 
program. Once equipped and experienced, the organization can also become a vendor of training services as well as 
provide simulator rentals without the responsibility and costs of providing training staff and materials. Other 
‘organizations may seek out such rental agreements to avoid the capital purchasing costs. 


‘Two additional considerations include team training and surgical assist training. Some of the increased length of 
robotic surgical cases is due to set-up time and docking times. Ensuring proper taining of surgical assists and 
‘nursing in addition to physician training may decrease these times and costs. 








Upper Limit 
‘There are upper limits to the improvements that can be achieved. For actual surgical procedures, these are driven by 
the number of robots available to perform surgery. Increasing surgeon availability beyond the capacity of the robotic 
OR will not provide improvements. An additional consideration is the availability of the robotic simulator for daily 
use or the training efficiency of the robotic simulator. If an institution chooses to invest in a DVSS trainer but does 
not have a dedicated training console, then training availability will be limited when compared to those with a 
standalone trainer, 


CONCLUSION 
In 1927, William Mayo famously stated that, “There is no excuse for the surgeon to learn on the patient.” In the 


ensuing decades, there have been many advances in the education of surgeons. Simulators offer the next 
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improvement along this path with capabilites that are very difficult to match through any other mode of training. 
We are entering a period in which competence-based learning is becoming the standard. This is an environment in 
which simulators possess undeniable advantages and empower the preparation of demonstrably competent surgeons. 


Robotic simulators are available but are not yet a standardized part of the training process. As the airlines and the 
military discovered in training pilots and combat personnel, simulation devices provide so many advantages in the 
learning process that they have become a mandatory part of those training curricula. The expectation is that the same 
will occur with robotic simulators over time. 


We recognize that many of the proposed financial and non-financial benefits are theoretical and that simulation in a 
‘well-designed curriculum purports different benefits than just the simulators themselves. There is an absence in the 
literature regarding the direct correlation between simulation and many of the positive impacts noted in this study. 

spite this absence, there is evidence of three things that make these impacts real: the evidence of transfer of 
‘raining with simulation as evidenced by validation studies, the presence of a steep learning curve (trends of up to 
100 cases before true competency is achieved), and the improvements in surgical outcomes with high volume 
sand centers. We also know that hospital systems and practices are highly variable and all factors cannot be 
ed and/or quantified financially. If there is not a direct return on investment, we are not suggesting that 
simulation based education is of no value. We designed the calculator to try to capture simulation effects on a 
business practice not only to justify the investment but to identify best practices and ways to implement robotic 
surgical curriculum in a meaningful and comprehensive way. These concepts should guide future research, 








‘To summarize, the use of simulators in training robotic surgeons requires an investment in equipment, staff, 
facilities, and supplies. But it also offers a return on this investment in surgeon productivity, surgeon health, hospital 
costs, and other training costs. The realizable returns will vary by institution based on the specifics of 
implementation and the existing ecosystem in which they are inserted. The ROI model in this report is one tool to 
assist in calculating the return that can be expected by organizations that make the investment in these devices and 
‘raining programs. 
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ABSTRACT 


‘The introduction of simulation into minimally invasive robotic surgery is relatively recent and has seen rapid 
advancement; therefore, a need exists to develop training curriculums and to identify systems that will be most 
effective at improving surgical skills. Several robotic simulators have been introduced to support these aims, but 
their effectiveness has yet to be fully evaluated. 





Currently, there are three simulators -- the daVinci Skills Simulator, Mimic dV-Trainer, and Surgical 
Systems’ RoSS. While multiple studies have been conducted to demonstrate the validity of each system, n 
have been conducted which compare the value of these devices as tools for education and skills improvement. 





‘This paper presents the results of an experiment comparing value, usability, and validity of all three systems. 
‘Subjects who were qualified as medical students or physicians (n=105) performed one exercise on each of the three 
simulators and completed two questionnaires, one regarding their experience with each device and a second 
regarding the comparative effects of the simulators. This data confirmed the face, content, and construct validity for 
the dV-Trainer and Skills Simulator. Similar validties could not be confirmed for the RoSS. Greater than 80% of 
the time, participants chose the Skills Simulator in terms of physical comfort, ergonomics, and overall choice. 
However, only 55% thought the skills simulator was worth the cost of the equipment. ‘The dV-Trainer had the 
highest cost preference scores with 71% percent of respondents feeling it was worth the investment. 











‘This work is the second component of a three-part analysis. In the previous study, the simulators were objectively 
reviewed and compared in terms of their system capabilities. The third part will evaluate the transfer of taining 
effect of each simulator. Collectively, this work will offer end users and potential buyers a comparison of the value 
‘and preferences of robotic simulators. 
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INTRODUCTION 


Robotic surgery has introduced a new dimension into the surgical field. With the introduction of robotic technology 
between patient and surgeon, a need fo master new skills has emerged. Medicine has come to the conclusion that the 
Halstedian training model (See one, do one, teach one) is no longer sufficient for teaching complex skills, especially 
robotic surgical skills (Cameron, 1997). A number of simulators have been developed to support training and skill 
assessment in robotic surgery. The currently available dedicated robotic simulators include: the da Vinci Skills 
Simulator (4VSS) by Intuitive Surgical Inc.. also known as the “Backpack Simulator”: the dV-Trainer from Mimic 
‘Technologies Inc.; and the RoSS by Simulated Surgical Sciences LLC (Figure 1). The purpose of these simulators is 
1o train surgeons prior to using the actual system and to allow them to acquire the necessary robotic skills to perform 
a safe surgery. All of these da Vinci simulators utilize a visual scene that is presented in a computer generated 3D 
environment providing challenging tests for practicing dexterity and machine operations. Originally, the simulated 
exercises trained basic robotic skills; however with advances in technology, surgeons can now train for specific 
procedures (e.g. nephrectomy and hysterectomy). 








dvss dV-Trainer RoSS 











Figure 1. Simulators of the da Vinel robotie surgical system 


‘Our hospital research laboratory has purchased each of these three simulators for the purpose of studying their 
effectiveness and applying them to the education of robotic surgeons, specifically for the Department of Defense 
(DoD). The DoD is interested in the effectiveness of the simulators to train military surgeons prior to and after 
returning home from deployments. This research is structured as three distinct stages. 


From the first stage of this work, the authors summarized the objective characteristics of the three systems. This 
included descriptions of the exercises offered in each, metrics used to evaluate students, overview of the system 
administration functions, physical dimensions and configurations of the equipment, and comparisons of the costs of 
the devices and their support equipment (Smith & Truong, 2013). In the first simulator, the trainee sits at and 
‘operates the simulated environment using the actual da Vinei surgical console. The simulator is a custom computer 
‘appended to the surgical console through the actual surgical data port. While the simulator costs approximately 
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$100,000, the surgical console costs $500,000 incurring an investment of $600,000. Using this simulator, users can 
‘ain using the actual hardware they would use during surgery: however, this requires the use of the surgical console 
that may be needed to conduct surgeries. Most hospitals may not have a dedicated training console, meaning that 
users would not have appropriate access to the simulator. The second is a standalone system that utilizes a 
‘eraphic/gaming computer, connected to a custom desktop Viewing and control device that replicates the hardware of 
the da Vinci surgeon's console. This system shares similar software with the dVSS, but does not require the use of 
any actual da Vinci hardware. The cost of this simulator is approximately $100,000. The third is composed of a 
completely customized replica of the da Vinci surgeon's console. Internally the simulator contains a graphic 
‘computer, a 3D monitor, and commercial Omni Phantom haptic controllers. This simulator uses unique software and 
is a litle more than $100,000 (Smith &Truong, 2013). 








‘This paper reports on the second stage of this research, in which the validity and usability of the simulators is 
‘examined. The third stage will be a measure of learning effectiveness using the systems, 





Surgical Simulation 


‘The validity of medical and surgical simulators is usually measured by the categories defined by McDougal (2007). 
‘This paper defines the most commonly recognized forms of validation as: face, content, construct, concurrent, and 
predictive validity. Face validity is typically assessed informally by users and is used to determine whether the 
simulator is an accurate representation of the actual system (i.e. the realism of the simulator). Content validity is the 
measure of the appropriateness of the system as a teaching modality. Experts who are knowledgeable about the 
device typically assess this via a formal evaluation. Construct validity is the ability of a simulator to differentiate 
between the performances of experienced users and those who are novices. Concurrent validity is the extent 10 
Which the simulator correlates with the “gold standard” and predictive validity is the extent to which the simulator 
can predict a user's future performance. Collectively, concurrent and predictive validity are known as criterion 
validity and are used as measures of the simulator's ability to correlate trainee performance with their real life 
performance. Face and content validity are most effective in evaluating the ability of a simulator to train a surgeon: 
however construct, concurrent, and predictive validity are most useful for evaluating the effectiveness of a simulator 
to assess a trainee. 





‘The validity of all three simulators has been tested and reported separately for the da Vinci skill simulator (Hung, 
Zebnder, Patil, 2011; Kelly, Margules, Kundavaram, 2012; Liss, Abdelshehid, Quach, 2012), the dV-Trainer 
(Kenney, Wszolek, Gould, Libertino, Moinzadeh, 2009; Sethi, Peine, Mohammadi, 2009; Lee, Mucksavage, Kerbl, 
2012) and the RoSS (Seixas-Mikelus, Kesavadas, Srimathveeravalli, 2010; Stegemann et al., 2013; Colaco, Balica, 
Su, 2012; Raza et al., 2013). To our knowledge only one publication has compared features of two of the simulators, 
but no comparative studies have been performed with all three of the systems (Liss MA, Abdelshehid C, Quach S. 
2012). Thus, the current study aimed to compare all three commercially available da Vinci simulators and detail the 
ndings for face, content, and construct validity for the three systems, 








METHODS 
Recruitment 


Participants in this study included medical students, residents, fellows, and attending physicians. Participants were 
recruited from the University of Central Florida Medical School, courses held at the Nicholson Center, and two 
medical robotic conferences (World Robotics Gynecology Congress and Society of Robotic Surgeons Scientific 
Meeting). Subjects were excluded from participating if they indicated that they had participated in a formal robotic 
simulation-training course. 





Each participant was categorized into one of three groups (i.e. Expert, Intermediate, or Novice) according to the self- 
reported number of robotic cases (ie. procedures) he or she had performed. Individuals performing 0-19 robotic 
cases in which they had 50% or greater console time were categorized as Novices, individuals with 20-99 robotic 
cases were considered to be Intermediates, and individuals with 100 or more cases were considered to be Experts. 


Materials 
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After being categorized into an experience level, each participant was assigned a specific order in which they used 
cach of the simulators (Figure 2). This order system was used to identify and potentially eliminate any bias that may 
exist by using a specific system first. All participants completed one exercise on each of the simulators. The tasks 
chosen were Peg Board | in both the dV-Trainer and the dVSS and Ball Placement 1 in the RoSS. The same task 
was used for both the dV-Trainer and the dVSS because these systems share similar software and exercises. The 
RoSS software contains unique exercises and Ball Placement 1 is designed to teach the same skills as Peg Board 1 


4 
ro $ 














Survey? 

















Figure 2. Rotating order of use by subjects, with survey order. 


‘After each exercise on each simulator, participants completed a post questionnaire (Survey 1), which asked for 
feedback regarding their experience on that specific simulator. After using all three systems, subjects completed a 
second post questionnaire (Survey 2), which asked them to compare all three systems to each other. The 
participant's performance metrics Were also collected from each of the simulators. 


RESULTS 
Demographics 


Subjects were categorized as Novice (n=37), Intermediate (n=31), or Expert (n=37). Sixty-two percent of subjects 
\Were men and 38% were women with an average age of 43. On average, participants had 15 years in practice and 3 
years of robotic experience. Seventy-six percent were attending physicians and 73% of participants were currently 
or had received robotic training, while 41% provided that they ain residents and fellows. There were differences in 
the average age and number of years in practice of participants based on the classification of expert, intermediate or 
novice (number of robotic procedures). These are to be expected, since higher ages are required to achieve higher 
raumber of years of practice and larger numbers of robotic procedures. 


Validation 
The types of validity evaluated in this experiment were face, content, and construct. To analyze the systems for face 


validity and content validity, questions from Survey I were used. The questions were evaluated on a five point 
Likert scale (Strongly Disagree, Disagree, Neither Agree or Disagree, Agree, and Strongly Agree). Face validity was 
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analyzed by expert and intermediate feedback as recommended by Van Nortwick et al. (2010) because these are the 
users most familiar with the robotic system; however, only expert feedback was used for content validity because 
they have the best ability to judge the appropriateness of the system as a training tool. For construct validity 
performance metrics such as Overall Score, Time to Complete, Number of Errors, and Economy of Motion were 
analyzed (Table 1), 














‘Table 1. Questions and data used for different levels of validity. 
Type of | Evaluation | Type of ‘Question/Metrie 
Valiait Participant 

Face | SuveyT | Expertand | Ql: The hand controllers on this simulator are effective for working 
Validity Intermediate | in the simulated environment (Likert) 





(Qa: The device isa sufficiently accurate representation of the real 
robotic system (Likert) 








Content] Survey 1 | Expert QE: The 3D graphical exercises inthe simulator are effective Tor 
Validity teaching robotic skills (Liker) 





‘QS: The scoring system effectively communicates my perfomance 
on the exercise (Likert) 





‘Qé: The scoring system elfectively guides me to improve 
performance on the simulator (Likert 





Construct | Simulator | Experts and | Overall Score (poms) 





Validity Novices ‘Number of Errors (count) 





Time to Complete (seconds) 





Economy of Motion (centimeters) 

















Face Validity 
‘The responses of Intermediate and Expert participants (n=68) were used to determine face validity (Table 2). A Chi 
square test of independence was used to evaluate the distribution of scores for a specitic simulator in relation to the 
‘order of the system's presentation to the subject. This analysis indicated that there Was no difference in participants 
answers according to the order in which the systems were presented; and established that no bias was present due to 
the presentation order (p>0.05). These questions asked participants to evaluate whether the hand controllers on the 
simulator were effective for working in the simulated environment (Question 1) and if the device is a sufficiently 
accurate representation of the real robotic system (Question 4). For both questions, the RoSS had the lowest average 
score, dV-Trainer had the second highest score, and the dVSS had the highest score of the three. A repeated 
measures ANOVA verified that the systems were scored differently for both questions (p<0.001). 





‘Table 2. Average scores from a S-point Likert scale on face validity. 





























DVSS__[aV-T Ross, 
‘Qi: The hand controllers on this simulator are effective for [4.80 3.02 217 
working in the simulated environment 
‘Qa: The device is a sufficiently accurate representation of | 565 35 12 
the real robotic system, 








Content Validity 
Expert (n=34) responses were used to determine whether the simulators were appropriate teaching modalities (Table 
3). As seen in Table 3, 100% of participants either agreed or strongly agreed that the 3D graphical exercises in the 
VSS were effective for teaching robotic skills while 59% disagreed or strongly disagreed that the RoS: 

capabilites were effective. When asked if the scoring system effectively communicated their performance, 88% of 
VSS users agreed or strongly agreed, while 79% of dV-Trainer users agreed or strongly agreed. Similarly, 91% and 
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82%¢ of participants agreed or strongly agreed that the dVSS and dV-Trainer, respectively, effectively guided them 
to improve their performance, while only 36% felt the RoSS provided the same guidance. 


‘Table 3. Scores on a $ point Likert scale for content validity questions. 












































Likert Score [Strong Dis | Disagree [Neither [Agree [Strong Agree 
[[2: The 3D graphical exercises tthe simulator are effective for teaching vobotic Skills 
DVSS 0% Of 0% 35.3% Tm 
aV-Tramer [29% 39% 118% 300% 2H 
Ross 206% BEI 17.6% 176% 39% 
‘OS: The scoring system effectively communicates miy performance on the exercise. 
DVSS 29% 5.9% 29% 382% 300% 
dv-Trainer [29% 29% 187% 350% Bo 
Ross 17.6% 20.6% 265% 29M 39% 
(06: The scoring system efectively guides me fo improve performance on the simulator. 
DVSS 0% Of BRE OLRM DH 
adv-Trainer | 29% 29% 118% CIR 206% 





Ros 





182% 182% Wat wae 30% 


























Construct Validity 
‘The overall score, number of errors, time to complete, and economy of motion scores collected by the simulators for 
Experts (n=37) and Novices (n=37) were used to compare construct validity (Table 4). Overall score is a metric 
synthesized by multiple metrics and is specific to the individual simulator, Intermediate subjects were not included 
in the construct validity analysis because it was only necessary to look if the simulator could distinguish specifically 
between novice and expert users. 





For the RoSS, the analysis has 23 missing data points because the system does not report scores when a user exceeds 
‘a maximum exercise time or chooses to terminate the exercise before completion. This resulted in a sample of 30 
experts and 21 novices on that system. A Mann-Whitney U test showed that the distributions of time (p=0.221), 
number of errors (p=0.644), and economy of motion (p=0.566) were not statistically different for the experts 
‘compared to the novice group. The overall score metric is not automatically exported by the simulator and therefore 
was not analyzed for this sytem, 


‘The dV-Trainer analysis of experts (n=37) and novices (n=37) had three missing values for economy of motion and 
completion time and five for the overall score mettic, thus the analysis contained varying number of subjects. A 
Mann-Whitney U test showed thatthe distribution of the overall scores was not significantly different for the expert 
compared to the novice group (p=0.061). These tests did confirm statistical differences for economy of motion 
(p<0.001) and time to complete (p<0.001) for this system with a lower economy of motion value and shorter 
‘completion time for expert users compared to novices, 











‘The dVSS analysis included all novice (n=37) and expert (n=37) participants. Using a Mann-Whitney U test, time to 
complete (p<0.001) and overall score (p=0.006) were significantly different for the expert compared to the novice 
‘group. The expert group had a higher score and a shorter completion time compared to the novice group. However, 
economy of motion did not show a statistical difference with this analysis (p=0.216). 
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‘Table 4. Mann-Whitney U test level of significance on construct validity measures 


























DVSS ‘aV-Trainer Ross, 
Time to Complete p=0.001 p<0.001 p=0.231 
‘Overall Score pz0.01 ala 
Economy of Motion p=0.216 p=0.001 p=0566 
‘Number of Errors ala nla p=0.644 

















‘The construct validity of the simulators was more specifically analyzed in terms of the self-reported number of cases 
of all participants (n=105) using a non-parametric correlation coefticient (Spearman's). For the RoSS, 30 
participants were excluded from the analysis. For the participants that were included in the analysis (n=75), there 
was not a significant correlation between time to complete (p=0.181), number of errors (p=0.563), or economy of 
‘motion (p=0.390) with the total number of robotic cases performed, 








For the dV-Trainer, four participants were excluded from the entire analysis and two participants were excluded 
from the overall score (Overall Score n=99; Economy of Motion and Time to Complete n=101). When analyzing the 
‘umber of participants’ robotic cases, there was a statistically significant correlation between overall score (p=0.03), 
economy of motion (p<0.01), and time to complete (p<0.01). The correlation value was negative for economy of 
motion and time to complete, showing that with a greater number of robotic cases, the time taken and distance 
moved decreased. The correlation was positive for overall score indicating that the participants’ score increased with 
the number of robotic cases performed, 











For the dVSS, two participants were excluded from the analysis (n=103). When analyzing the metrics in terms of the 
total number of robotic cases performed, there was a statistically significant difference between overall score (Pp 
=0.01) and time to complete (p <0.01). The correlation value was negative for time and positive for overall score, 
signifying that with more robotic cases the time taken decreased and the score increased. There was not a 
statistically significant correlation between economy of motion and the total number of robotic cases performed 
(p=0.108). 





‘Table 5. Correlation between level of experience and simulator scores 






































DVSS ‘aV-Trainer Ross, 
‘Overall Score =0.001 ‘031 ala 
Time to Complete p=0.001 p<0.007 P=o1sT 
Economy of Motion p=0.105 p=0.001 p=0.390) 
‘Number of Errors ala ala p=0503 





Usability (Preference) 


‘The questions from the Survey 2 were used to understand the preference of the subjects when using the simulators 
{All subjects were included in this analysis except for two participants who were dropped from the analysis because 
they did not complete the questionnaire. The participant's responses to the usability questions can be seen in Figure 


If you are (were) a program director, which simulator would you choose for your trainees 
In which simulator were you physically more comfortable: 

Which simulator had the best hand controls 

Which simulator had the best foot controls 

Which simulator had the best 3D vision: 

Were you feeling stressed or annoyed by any of the simulators? 
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Figure 3. Description of usability responses 


‘Overall, most participants preferred the dVSS and indicated that they would choose this device as a training system 
if they were a program director. Participants not only felt most comfortable in the dVSS, but also felt thatthe system 
‘had the best control and vision equipment. The least preferred system was the RoSS which most participants also 


‘agreed made them feel stressed 


‘or annoyed. Ten percent of participants also responded that they felt stressed or 


‘annoyed by both the dV-Trainer (dV'T) and the RoSS. 


Cost 


All participants were also asked to provide feedback on their simulator preference in terms of the cost of the system. 
‘The responses were analyzed in terms of the frequency of the responses given. Most participants felt that the mimic 
dV-Trainer was worth the investment; while most felt that the RSS was not worth the money. When asked about 
the dVSS, only 56% of participants agreed that it was worth the investment. Figure 4 provides a full description of 


the responses. 





avss, 
($600,000) 











Worth the Investment? 


Yes 
No 


_ +The price presented in the survey 
dV-Trainer — RoSS AInctuded all necessary support 


($100,000) ($100,000) euipment to make the sonalator 
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DISCUSSION 


‘The aim of this study was to conduct a comparison of the three commercially available simulators used to ain 
surgeons on the daVinei robotic system. The study was performed for the US Army to assist them in making a 
purchasing and deployment decision regarding robotic simulators. Their interest is in re-training robotic surgeons 
‘who have been deployed to combat zones, where they have served as trauma surgeons for many months. Prior to 
resuming their robotic specialties, these surgeons need a program to both refresh and re-validate their robotic skills. 
This study provided information about the face, content, and construct validity as well as usability of the systems. 
‘The simulators were perceived to be different in their representation of the real robotic system. The dVSS was most 
preferred in terms of ergonomics and usability; however, most participants did not feel that this system was worth a 
‘$600,000 investment. In terms of cost, most participants agreed that the dV-Trainer had the best cost-effectiveness. 
‘The RoSS was the least preferred system for comfort and other usability aspects (ie., hand controls, foot controls, 
and 3D interface), with most participants feeling stressed or annoyed when using the system. This study was unable 
to validate the face, content, or construct validity for this system, 








‘The dVSS leverages the actual hardware used to perform robotic surgeries for use in the simulated environment, 
which allows for a more realistic experience, but decrease its availability and creates a higher cost for training than 
‘other robotic simulators. Economy of motion was not able to differentiate novices from experts in the dVSS, which 
could be attributed to the ease of use of the controllers allowing novices to move the controls as efficiently as 
experts. The generous workspace of the dVSS could also have an impact on the lack of difference. In contrast to the 
VSS, the dV-Trainer is standalone simulator and does not require the support of the daVinci hardware to operate. 
‘This allows for beter accessibility and requires less of an investment for training. The overall score aspect of 
construct validity may not have shown a difference between novices and experts because of the way that the scoring 
is developed. The scoring system is constructed with a “ceiling” that prevents users from achieving a high overall 
score without attaining high scores across multiple metrics. 





Currently, there is limited data available that confirms construct validity of the RoSS. Similarly to Raza (2013), this 
study was unable to confirm a difference between experts and novices in terms of time taken to complete the 
exercise. Time to complete, as well as economy of motion, is considered a highly relevant measurement of expertise 
levels for robotic surgeons (Perrenot, Perez, Tran, Jehl, Felblinger, Bresler, & Hubert, 2012). To our knowledge this 
three-part study is the first to compare all three available systems. This study involved the largest sample size and 
diversity of participants (i.e., experience levels, number of robotic cases, and subspecialty type) thus far in relevant 
publications. The lack of consistency in the available exercises and scoring systems across the three systems was a 
limitation to the study. Considerations for future research would be to use more complex exercises and increase the 
depth of the face and content validity evaluation, 





‘Current research is focused on the effectiveness of the simulators and objectively measuring the ansfer of taining 
to the actual robotic system. All three simulators will be examined in this final stage of the experiment; however, the 
results of this three-part study will guide the choice of simulators used for future studies at Florida Hospital 
Nicholson Center and may also influence decisions at other laboratories. Also, this research may impact the 
purchasing decisions of customers for these devices. 
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Poster Title: 


Fundamentals of Robotic Surgery Psychomotor Skills: Metrics Development and Evaluation 





Introduc 


Robotic surgery has been established as an innovative approach in surgery due to a telemanipulator 
device, which introduced a new dimension into surgical tools. This device allows surgeons to manipulate 
robotic arms from a remote console to perform complex surgical procedures. Robotic surgical systems 
‘overcome laparoscopic limitations and facilitate the performance of minimally invasive surgery due to 3D 
vision, 7-degree-of-freedom instruments, tremor abolition, motion amplification, and stabilization of the 
camera (Patel et al, 2013; Hubens, Coveliers, Baliu, Ruppert, & Vaneerdeweg, 2003; Blavier, Gaudissart, 
Cadiére, & Nyssen, 2007). The system also offers 10x magnification, wristed instruments, and a third 
working arm. Currently, the only system is Intuitive’s da Vinci Surgical System. 


Robotic surgery has demonstrated safety and effectiveness for urologic, gynecologic, ENT, and complex 
general surgery procedures (Barbash, Friedman, Glied, & Steiner, 2014; Serati et al., 2014; Maan, Gibbins, 
AlJabri, & D'Souza, 2012; Luca et al,, 2013; Zureikat et al,, 2013). Exponential growth of minimally 
invasive procedures, particularly robotic.assisted procedures, raises the question of how to assess robotic 
surgical skills. This device also introduces a specific need for training and certification to ensure a minimal 
standard of care for all patients. Some institutions have attempted to develop and validate robotic 
training in regards to specific specialties (Chitwood et al,, 2001; Geller, Schuler, & Boggess, 2011; Grover, 
Tan, Srivastava, Leung, & Tewari, 2010; Chowriappa et al., 2014; Jarc & Curet, 2014); however, the lack of 
a national standard has pushed surgical societies (e.g. SAGES, SRS, and MIRA) to develop a unified 
approach and standard for robotic skills training Zorn et al., 2008). 


To develop a comprehensive model for robotic surgery, the Department of Defense, Veterans 
‘Administration, and fourteen surgical specialty societies convened multiple consensus conferences to 
‘create the Fundamentals of Robotic Surgery (FRS). A similar education and training initiative was 
implemented for use in laparoscopic surgery, which resulted in a curriculum called Fundamentals of 
Laparoscopic Surgery (FLS). FRS Conference participants included more than 80 subject matter experts 
(SMEs), consisting of surgeons, psychologists, engineers, simulation experts, and medical educators 
(Smith, Patel, Chauhan, & Satava, 2013). 


The committee's vision of FRS was driven by two main goals: to ensure a perfect understanding of the 
basics of robotic surgery and to develop a psychomotor skills program that focused on basic robotic tasks. 
The intended users for this program are novice robotic surgeons, who could be residents or fellows and 
attending surgeons who have never used the robotic system. Two assessment tools were created: an 
online curriculum for knowledge and team training skills and a device for psychomotor skill training and 
evaluation (Levy, n.d.). For the psychomotor skills portion of the training, physical and virtual devices 
‘were developed (Figure 2). 


«image of physical and virtual domes> 


Objective: 


The purpose is to describe the development of the exercise metrics and how they are designed to take 
into account the specificity of robotic errors. 
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Method: 





‘The members of the consensus conferences worked together to outline outcomes measures and metrics, 
which touched on the essential cognitive, psychomotor, and team training skill. This resulted in a 
prioritized matrix of 25 robotic surgery concepts, which is the core material used in the design and 
development of the FRS Curriculum. Of those 25 concepts, 16 are directiy linked with psychomotor skills 
(Table 1). The FRS committee members then identified seven exercises that incorporated all 16 skills. 



































Phase tems 
Pre-Operative Docking Robotic Trocar 
Energy Sources Clutching 
Camera Control Wrist Articulation 
Instrument Exchange Multi-Arm Control 
7 Eye-+hand coordination Atraumatic Handling 
Intra-Operative 
Dissection (Fine and Blunt) Cutting 
Needle Driving Suture Handling 
Knot Tying 
Post-Operative Undocking 











These exercises include docking and instrument insertion, tower transfer, knot tying, railroad track, 4th 
arm cutting, puzzle piece dissection, and vessel energy dissection (Table 2). Docking and instrument 
insertion is an essential and unique robotic skill to begin a procedure. Failure at this stage of the 
procedure can compromise the surgery. Tower transfer is a non-surgical exercise that introduces the 
utilization of endowrist manipulation and the 7 degrees of freedom to surgeons. Knat tying and railroad 
track are the base of a suturing exercise. The technology introduced in the wristed instruments facilitates 
the performance of these tasks. 4th arm cutting is another task specific to robotics, which improves 
surgeon’s autonomy. The 4th arm allows surgeons to manage three instruments by using a foot pedal to 
switch between working arms. Puzzle piece and vessel energy dissection are critical tasks, which 
incorporate complex articulation of instruments and application of energy (Le. cauterization and cutting) 








Exercises Skills 





Task1: Docking & Instrument Insertion: = Docking, 
1 - Instrument insertion 
i ~ Eye-hand coordination 


- Operative field of view 





Task 2: Ring Tower Transfer: + Eye-hand coordination 
= Camera navigation 

= Clutching 

- Wrist articulation 

= Actraumatic handling 
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Task 3: Knot Tying: 


+ Knot tying 

+ Suture handling 

~ Eye-hand coordination 
- Wrist articulation 





Task 4: Railroad Track: 





Needle handling & manipulation 
- Wrist articulation 

~ Actraumatic handling 

+ Eye-hand coordination 





Task 5: 4" Arm Cutting: 





= Multiple arm control & switch 
~ Cutting 

- Actraumatic handling 

- Eye-hand coordination 





Task 6: Puzzle Piece Dissection: 








Sharp and blunt dissection 
~ Cutting 

 Actraumatic handling 

+ Eye-hand coordination 

- Wrist articulation 





Task 7: Vessel Energy Dissection: 




















Energy sources use 
Sharp dissection 

= Cutting 

= Multiple arm contral 

- A-traumatic handling 

+ Eye-hand coordination 





After establishing the exercises, metrics were created to assess the users task specific performance. The 
development of these metrics began early in the device development process. To do this, the team 
consulted with experts robotic surgeons to determine what would constitute performing each exercise 
correctly. Often this required looking at the purpose of the exercise and what skills are being taught. By 
doing this, the team was able to outline metrics that were relevant to the task and important to robotic 
skills. We then evaluated how certain mistakes, which are specific to the FRS exercises, are translatable to 
robotic skill. For example, in ring tower transfer and knot tying, excessive instrument farce can be 
‘evaluated in the form of knocking a tower off of the surface. Other metrics like the time to complete the 
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task, the number of times the instruments are placed out of view, and the number of instrument 
collisions were also considered in the development. 


After the basic metrics were established, the team evaluated an expert robotic surgeon and a novice using 
the metrics. The expert helped the team to envision how the exercises should be performed, while the 
novice testers helped to see what mistakes are typically made by inexperienced users. The metrics were 
also tested using different types of materials with different attributes to ensure that changes in the 
materials would not limit the user's ability to achieve the desired metrics. For example, it was important 
that the synthetic fat material have a slightly sticky texture to evaluate the users performance in 
dissecting the skin layer from the fat. 


Once established, it was important to consider how the metrics would be measured, Since the scoring 
‘would be performed via expert video review, the research team had to consider the feasibility of the 
scoring for the reviewer. For example, it was not realistic to ask the reviewer to account for the seconds 
that the instruments were out of view or attempt to measure the number of centimeters the user cut 
‘outside a designated line. So, for these metrics it was decided to use a count or a yes/ no type response, 
Also, some metrics were added to ensure that the exercise was performed correctly. For example, in 3° 
‘arm cutting exercise, a metric was added to ensure that the user performed the cuts in the correct order. 
This is relevant because when performed in another order, the user does not actually have to practice 
‘switching back and forth between arms. 


After the physical dome metrics were established, they were given to simulation companies to use as 
metrics for evaluating the simulated dome exercises. When working with the simulation teams for 
development of the virtual domes, it was important to ensure that the virtual domes mimicked the 
physical properties of the actual dome. This was highly important because the simulations may be used 
‘without a proctor and the physical behaviors have a considerable impact on the scoring metrics and 
‘guidance that is given for improving performance. The research team worked with simulation developers 
to evaluate the simulated exercise properties including elasticity of materials, flexibility of sutures, 
simulated gravity, and the effects of excess force on the virtual device to ensure that it behaved similar to 
the real dome. Thus, creating similar metrics across all dame modalities. Many “what if" situations were 
‘examined to determine how they would be handled in the real dome and how the simulation should 
handle them. These incidences could have different implications for instruction and assessment in a real 
dome exercise, as apposed to in the simulated environment. Finally, it was an important consideration to 
leverage the capabilities of the simulated environment to objectively assess the users performance, 


Results: 


The physical FRS dome is scored using a validated global assessment called the Global Evaluative 
‘Assessment of Robotic Skills (GEARS). This looks at the overall performance of the user in terms of their 
robotic skills. The described work resulted in task specific metrics for each exercise (Table 3) 

<insert metrics table> 


Most of the metrics used to evaluate the virtual dame are similar to the metrics of the physical dome. 
However, some of the metrics do vary between the physical and simulated domes. While the physical 
dome is scored via expert video reviewing, the simulator can more objectively assess a user's 
performance. This allows the simulated exercises to score some errors more accurately, such as 
instruments being out of view for a specific amount of time (seconds) and over a specific distance 
(centimeters) 


Conclusions: 
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Many considerations were made during the development of the FRS psychomotor skills metric. From this 
process, we have taken away several lessons that helped to develop our metrics including: 

Perform preliminary testing with varying expertise level users 

* Create specific and measurable standards for assessment 

Work closely with the simulation development teams for the details of simulation 

* Constant testing and expert consultation 


twas important to determine how users of varying expertise level would use the device, including 
determining how the expert would perform it and what mistakes are typically made by novices. t was 
important for us to ensure that the metrics developed were specific and measurable under the 
circumstances that the users would be assessed. The video reviewing did not allow for certain measures 
to be assessed causing some metrics for the physical dome to be modified to an easier format for video 
review. The simulated environments however allow for a much more objective assessment and it was, 
beneficial for our team to work closely with the simulation developers to establish what metrics might 
‘change in the virtual modality. Finally, it was very important to solicit expert feedback on the metrics 
throughout the process. 


Further validation testing is necessary for the FRS curriculum. This is currently being assessed via a pilot 
study and eventually in 2 multi-site validation study. The goal ofthe pilot study is to establish preliminary 
scoring benchmarks based on expert performance, which will be used to guide the multi-site validation 
study. 
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Robotix Mentor (3D Systems/S imbionix) 





Part: System Functionality Comparison 


Features 


“dV-Trainer 


Ross, 





[System Manufacturer 


Mimic Technologies Inc 


[Simulated Surgical Systems LLC 





Specifications 
(Simulator only) 


eight 25" Width 23° 
120 oF 240V power 


Depth 36° Height 26° Width 43 
120 o 240V power 


Depth 44° Height 7 Widih 45" 
120 or 240V power 





[Specifications 
(Complete System as shown in Figure 1) 


[Depth 40" Height 6" Widih 40" 
120 of 240V power 


Depth 36° Height 50” Width 5° 
120 oF 240V power 


Depih 44° Height 77° Width 43" 
120 or 240V power 





Visual Resolution 


VGA 1024 768 


VGA 10d x 168 


VGA 0x 480 





[Components 


[Customized computer aflached to da Vinci 
surgical console 


Standard computer, visual system with 


hhand controls, foot pedals, 


Single integrated Simulation 


device 


custom 





[Support Equipment 


ida Vinci Si surgical console, custom data 
cable 


Adjustable table, touch screen monitor, 
keyboard, mouse, protective 
custom shipping container 


USB adapier, Keyboard, mouse 





[Exercises 


[40 simulation exercises 
(35 by Mimic, 5 by Simbionix) 


65 simulation exercises 


[52 simulation exercises 





[Optional Software 


[PC-based Simulation management 


share curriculum sharing web ile 


Video and Haplies-based Procedure 


Exercises (HoST) 





[Scoring Method 


[Sealed 0-100% with passing thresholds In 
multiple skill areas 


Proficiency-based point system with 
passing thresholds in multiple skill areas 


Point sysiem with passing thresholds In 
multiple skill areas 





[Student Data Management 


[Custom control application for external 
PC. Export via USB memory stick. 


Export student data to delimited data fle 
and graphical reports. 


[Export student data to delimited data file, 





None 


Select any combination of exercises. Set 
passing thresholds and conditions. 


Select specifically grouped exercises. Set 
passing thresholds. 





[Administrator Functions 


[Create student accounts on external PC 
Import via USB memory stick. 


Create student 
curriculum, 


‘accounts, Customize 


(Create student 
curriculum, 


‘accounis, Customize 





System Setup 


None. 


Calibrate controls 


[Calibrate controls 





[System Security 


[Student account ID and password 


PC password, Administrator password 
Student account ID and password. 


PC password, Administrator password, 
Student account ID and password. 





[Simulator Base Price 


585,000} 


39.200 


126,000] 





[Support Equipment Price 


'5500,000] 


39.800 


30 





Total Functional Price 





S585 000] 





3109,000 





3126.00] 





Usability Value 
RoSS 
2% 


dV-Trainer 
8% 


a | ovss 
a | | dV-Trainer 
| 





Simulators 





Core Skills Content| dV-Trainer Ross Dvss 
Wristed Bimanual Tasks Yes Yes Yes 
Manipulation Metrics No No No 





Maneuver Camera Tasks Yes Yes Yes 
Metrics No No No 


Master Cntrl Clutching Tasks No Yes Yes 
Metrics | M Wrk Rng No M Wrk Rng 


Use 3rd Arm Tasks Yes Yes Yes 
Metrics No No No 


Depth/3D Perception Tasks No No No 
Metrics No No No 


Aware Instrument 
Force Tasks No No 
Metrics | Ex Inst Fre Ex Inst Fre 
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Core Skills 


Content 


Simulators 





dV-Trainer 


RoSS 


Dvss 


Robotix 
Mentor 





Wristed Bimanual 
Manipulation 
Maneuver Camera 


Master Cntrl Clutching 


Use 3rd Arm 


Depth/3D Perception 


Aware Instrument 
Force 


Tasks 
Metrics 


Tasks 
Metrics 


Tasks 
Metrics 


Tasks 
Metrics 


Tasks 


Metrics 


Tasks 
Metrics 





Yes 
No 
No 


No 
M Wrk Rng 


Yes 
No 


No 


No 


No 
Ex Inst Fre 





Yes 
No 


Yes 
No 


Yes 
No 





Yes 
No 


Yes 
No 


Yes 
M Wrk Rng 


Yes 
No 


No 


No 


No 
Ex Inst Fre 





Yes 
No 














Simulator Investment Costs 










*Cost of Simulators 
«Annual Maintenance Fee . 
Facility Costs 
Staffing Costs 
«Supplies & Materials 


Supolies & Materials 
Investment 














Surgeon Productivity 
Number of Surgeons Trained 
Mean Length of Operation 
‘Time Saved per Procedure 
Length of surgical Day 


hours 
minutes 
hours 


Surgeries per day Fr 
New Free Surgery Time per day 00 hours 

‘Added Surgeries per Dey 1 

Revenue per surgery 

‘Adsitional Revenue per day $10,000 

Surgical Days [ES atowloerveor __| 
‘Additional Revenue pervYear $1,000,000] $1,000,000 















Surgeon Productivity 

«Reduced Surgery Time 

«Increased Surgeries per Day 
«Surgeon Health 

«Increased Surgeon Stamina (hrs/day) 


Hospital Costs 

*Reduced Liability Insurance 
«Reduced Instrument Breakage 
«Reduced Surgical Error Rate 


Other Training Costs 

*Reduced Training in the OR 
*Reduced In-house Instructors 
*Reduced External Training w Travel 





extemal Training Event cost (w/ travel) 
Number of External Events 














Robotic Simulator RO| Model 


$277,200] 


Year 1 Investment|Year i Retum | Year 1 ROI Year 1 Pay Back 


Year 2tnvestment|Year2Return _|Year 2ROL Year2 =o: Back 
$i.izs,1so]__1s74%] _19] 
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Fundamentals of Robotic Surgery 


The Fundamentals of Robotic (FRS) is a joint educational 
program funded through a Department of Defense grant and 
an unrestricted educational grant from Intuitive Surgical. 

FRS is a multi-specialty, proficiency-based curriculum of basic 
technical skills to train and assess surgeons to safely and 
efficiently perform robotic-assisted surgery. 


It was developed by over 80 national/international robotic 
surgery experts, behavioral psychologists, medical educators, 
statisticians and psychometricians. 

The clinical robotic surgery subject matter experts 
represented all of the major surgical specialties in the United 
States that currently performs robotic-assisted surgical 
procedures, the Department of Defense and the Veterans 


Administration (VA). 
@FRS 


Richard Satava, MD 
Professor of Surgery 
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Leadership 





Robert M. Sweet, MD 
Roger D. Smith, PhD Vipul R. Patel, MD Associate Professor Urology 
nology Officer Medical Drector, 











Genter jobal Robotics institute 


on, FL londa Hospital Celebration Healtn 





Dimitrios Stefanidis, MD, Amold Advincula, MD Martin A. Martino, MD 
PhD Professor of Obstetrics & Medical Director 
Medical Director Gynecolagy Minimally Invasive Robotic Siagery 





slumbia University Lehigh ath Network 





College of Physicians and Allentown, PA 


SERS ena § 





Organizations Represented 


American Academy of Orthopedic 
Surgeons 

American Academy of Otolaryngology — 
Head and Neck Surgery 

American Association of Gynecologic 
Laparoscopists 

American Association for Thoracic Surgery 
American Board of Surgeons 
American Board of Urology 

American College of Surgery 

American College of Obstetricians and 
Gynecologists 

American Hernia Society 

American Society of Colon & Rectal 
Surgeons 

American Society of Plastic Surgeons 
American Urological Association 
American Urogynecologic Society 
Arthroscopy Association of North America 
Asia Pacific Hernia Society 
Association for Surgical Education 
China Hernia Society 

Latin American Association of 
Laparoscopic Surgery 


Latin American Hernia Foundation 
Minimally Invasive Robotic Association 
National Institute for Health Research, UK 
Resident Review Committee — Surgery 
Resident Review Committee — Urology 
Royal Australasian College of Surgeons 
Royal College of Surgeons, Ireland 
Society of American Gastrointestinal and 
Endoscopic Surgeons 

Society of Laparoendoscopic Surgeons 
United States Department of Defense 
Veterans Health Administration, National 
SimLEARN Center 
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Setting the Benchmark Criteria for Any Curriculum 

Train expert/experienced surgeons to their learning curve 
Two consecutive trials with no improvement 

Calculate the mean of their performance 

Calculate the standard deviation for other levels 
Proficiency is the Mean of Expert Performance 





Summary 


* Validation Trials are underway 
— Planned Completion in October 2015 
* Online Curriculum and psychomotor device are 
openly available now 
— FRSurgery.org 
— FRSDome.com 
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INTRODUCTION 


Creating a viable robotic surgery practice within a hospital is an expensive and risky endeavor. The investment in equipment, facilities, personnel, and process modifi- 
cation is significant, amounting to millions of dollars in the early years of a program. Many hospitals make this investment with a limited understanding of how best to 
structure a robotics practice and the probability of achieving a positive return on this investment. At the business end of a decision to create a robotic surgery prac- 
tice, the hospital has the goals of optimizing the utilization of the robotic operating room, reducing costs, and ensuring patient safety. Training to and maintaining the 
competency of the surgeons performing the procedures has a direct impact on these areas, where simulators can make valuable contributions in surgeon perfor- 
mance and productivity. The potential business effects of these devices are summarized in Figure 1. The inclusion of simulators in the design of a robotic surgery 
practice and the training necessary to develop and maintain skills over multiple years can have an impact in all of these areas. Given the appropriate template for dis- 
cussion and a tool to assist in calculating the benefits, business and clinical leaders should be able to arrive at a more accurate and objective decision regarding the 
purchase and use of simulator devices as part of a robotic surgery program. 


Figure 1. Summary of Simulation Effects on Robotic Surgical Practice 
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Surgeon Productivity Training Costs 
Surgeon Stamina OR Costs 
Surgeon Competence Medical Errors 
Surgeon Certification Instrument Breakage 
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Robotic Surgery Simulation 


For every complex and expensive system there emerges a need for training devices and scenarios that will assist new learners in mastering the use of the device and 
understanding how to apply it with value. Intuitive Surgical’s da Vinci robot is just such a system. It is currently the only FDA approved device for laparoscopic robotic 
surgery on human patients. Despite the 1.5 to 2 million dollar price tag, the device has seen rapid distribution and its implementation has led to the need to develop 
more efficient and effective training methods, as well as assessment and skill maintenance tools. In laparoscopic surgery, simulators have played an important role in 
improving the practice of surgery over the last 20 years +”. The same trends and values will likely apply in robotic surgery. 


The complexity, criticality, and cost associated with the application of the da Vinci surgical robot have stimulated the commercial creation of simulators which repli- 
cate the operations of this robot. There are currently four different simulation systems available for training and developing skills in robotic surgery: da Vinci Skills 
Simulator (Intuitive Surgical Inc.); dV-Trainer (Mimic Technologies, Inc.); RoSS (Simulated Surgical Skills LLC); and Robotix Mentor (Simbionix Ltd). Each of these pos- 
sesses unique traits which make them valuable solutions for different types of users and learning environments. Investment in the simulators alone represents a ma- 
jor capital investment as costs range between $85,000 and $125,000 per device. Coupled with increases in direct costs from operating room time and supplies, ro- 
botic surgery implementation can have a profound impact on hospital finances **°°’. These do not include costs associated with ensuring patient safety and sur- 
geon training. From a business perspective, a hospital, college of medicine, robotic practice, or education center should calculate a financial return that will be 
achieved through the purchase and use of these devices. 


Simulation and ROI 


Rehearsal and simulation has been one of the primary means of developing and maintaining proficiency in specific skills for thousands of years. Lectures and written 
materials are the primary means for developing cognitive knowledge, but these are not effective at instilling psychomotor skills in any field. The skills needed in the 
hands, body, and coordination with the mind must be developed through practice. Rehearsal in a non-lethal environment has become the standard of practice for 
learning in military warfare and aviation. The lives of soldiers and pilots today are considered of significant value to demand a structured training program and meas- 
ure of proficiency before entering a life threatening situation. Given the technology that has emerged in the late 20th and early 21st centuries, the means of rehearsal 
have shifted from drills to events that are mediated by computers with the ability to measure performance and provide constructive feedback to improve perfor- 
mance. Implementation of direct practice on simulators has led to significant quality improvements and a reduction in training costs to the tune of hundreds of mil- 
lions of dollars. There is a 30 to 40 year history of success documented in the military literature, specifically regarding its return on investment. On average simula- 
tors cost only 5-20% of live training and can reduce the length of training by 10-30% = 


Simulation devices are similarly a prominent tool in medical and surgical education. From the most primitive stuffed dolls with anatomical markings used in ancient 
Chinese societies, to the most current computer driven, three dimensional representations of living tissue, simulators have been helping to train surgeons for over a 
thousand years. While there are parallels between the military and medical fields regarding the risk to human life and the need for structured high fidelity training, 
there are also unique challenges in medicine. There are many confounders in medicine and the seemingly direct relationship between simulation and improved pa- 
tient outcomes and reduction in costs is not robust. So, while most agree that simulation is a necessary requirement for surgical training, the lack of evidence cou- 
pled with the costs of devices and curricula development have led to a slower adoption of simulation in surgical education when compared to other industries. One 
of our goals is to use the lessons from our military and aviation industries in conjunction with published literature to identify key variables that impact the ROI of ro- 
botic surgical simulators and provide a loose framework for how simulators can be better utilized to achieve financial, educational, and patient safety benefits. We will 
focus on the concept of return on investment (ROI) for these simulators, not on the capabilities of any one specific device. Our goal is to assist the purchasers and us- 
ers of these devices in determining whether such a device is a sound financial investment. 
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METHODS 


The first step in developing a model was to identify, at a basic level, the main effects that robotic simulation would have on a healthcare organization. We found that 
the use of a simulator can impact at least three major, but separate parts of the healthcare delivery process (Figure 2). 
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Figure 2. Robotic Surgery Simulator Impact Areas 


The first, and most important, is patient outcome. The use of a simulator can potentially improve patient outcome by allowing the surgeon to perform more effective- 
ly and efficiently with fewer adverse events. 


The second is the impact on the surgeon. He or she may find that the ability to use the robot effectively impacts their career with additional options that can be 
offered to the patient and to potential hospital employers for the surgeon. It may also reduce the long-term wear and tear on the surgeon’s body due to improved er- 
gonomics, making it possible to continue practicing surgery for a longer period of time. 


Third is the effect that it will have on the costs and efficiency of the business practice. Simulator trained surgeons may perform procedures more rapidly, with less er- 
ror, and lower instances of equipment breakage. This also contributes directly to improved patient outcomes. 


Literature Review 


There are very few studies evaluating ROI for the simulators of the robot. There is one article directly evaluating the RoSS simulator where annual training hours were 
converted to training time on a robotic console *. In that study, the use of the stand-alone simulator resulted in cost savings of $600,000. Animate lab training of the 
same duration would have cost approximately $72,000 annually to train 100 people each year. The remaining the studies look solely at laparoscopic simulators. For 
example, in March 2004, Frost & Sullivan Inc. conducted an ROI study on three training simulators sold by Immersion Medical. Specifically, the data regarding the Lap- 
aroscopy AccuTouch System was analyzed. The Laparoscopy AccuTouch System uses advanced 3D technology and graphics to re-create the procedures and environ- 
ment of abdominal laparoscopic surgery. Using reported median values from survey data, financial benefits were estimated at $168,767, based on annual costs of 
$76,000, with an estimated payback period of 169 days. Also somewhat related, is an article by Leddy et al that reviewed the published cost analyses of robotic sur- 
gery for both urology and pediatric surgery, in an effort to develop a novel model for determining return on investment °°. The premise for the model was that reduc- 
tion in costs directly relate to the ability of the technology to reduce hospital length of stay. Surgical volume can be limited by the availability of hospital beds, in 
which shorter hospital stays lead to greater bed availability, which can open space so that more surgical procedures can be performed. It also acknowledged the 
learning curve of robotic surgery and the expectation for OR times to decrease with improved technology, surgical technique, and time. 


Systems Analysis 


With this as a foundation, we used a systems analysis method popularized by Jay Forrester at Massachusetts Institute of Technology (MIT) to explore the interrelation- 
ships between a large number of variables within the complex and dynamic hospital surgical program. This identified variables which react almost immediately to 
changes in the training offered to robotic surgeons as well as longer-term variables which may take years to emerge as measurable returns on investment. Using 
those systems diagrams we distilled and grouped variables and relationships into the model shown in Figure 3. Note that each variable is associated with a calculable 
financial outcome. This model was useful in identifying the factors which contributed to ROI at a first, second, and third level of separation. It also motivated discus- 
sions around the financial and non-financial variables that were changing due to the incorporation of simulation. 
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Figure 3. Effects of Simulation-based Training on Robotic Surgery Business 


Expert Opinion 

Twelve expert interviews were conducted at several surgical and simulation centers around the country and internationally to further develop the model, including 
identifying and quantifying new variables. Surgical leaders interviewed included Randy Fagin MD, Texas Institute of Robotics; Brendan Sayers PA, University of Texas — 
Southwestern; Arnold Advincula MD, Celebration Health; Thomas Lendvay, MD, University of Washington College of Medicine; Robert Sweet, MD, University of Min- 
nesota College of Medicine; Col Timothy Brand, MD, Madigan Army Medical Center; Jacques Hubert, MD, University of Lorraine Medical School; Dimitrios Stefanidis, 
MD, Carolinas Healthcare System; Martin Martino, MD, Lehigh Valley Health Network; Mona Orady, MD, Cleveland Clinic; John Lenihan, MD, MultiCare Health Sys- 
tems; and Michael Pitter, MD, Newark Beth Israel Medical Center. 


There were several common themes identified from these expert interviews. Most agreed that having a defined curriculum was imperative to successful training and 
that it should be identified prior to purchasing a simulator. These robotic training curriculums consisted mostly of manufacturer-developed didactic education with a 
combination of simulator and/or dry lab assessment. It was also clear that there were a variety of successful arrangements for location and proctoring of a robotic 
simulation curriculum. The majority utilized a combination of the DVSS (Intuitive Surgical Inc.) and the dV-Trainer (Mimic Technologies Inc.). Simulation was most 
often a part of resident and fellow curriculums but approximately 50% of the sites also had attending level programs. Simulator value was also tied directly to the ac- 
cessibility of the equipment depending on its placement in the facility. 

They verbalized a strong need for studies evaluating correlations between simulation and improved clinical outcomes, determining the ideal length and intervals for 
simulation training, and comparative studies between institutions with and without simulation centers. Curricula were called for in maintenance and certification, as 
well as remedial training following complications. The surgeons also showed a shared interest in the development of procedure-specific training and increasing com- 
plexity of the simulated exercises. Several believed that simulation could be used globally to decrease insurance costs and increase reimbursements. 


RESULTS 

Using the information in the system dynamics diagrams and the interviews with the surgeon educators, we created a basic mode! of ROI focused on short-term, 
measurable returns. The model is populated with general numbers from the literature and the authors’ institutions, but the user is able to enter specific values repre- 
senting their unique facilities. Users enter data for their institutions and the model calculates intermediate values and an annual return for Years 1 & 2 following the 
investment. In most cases, years three and later represent an ROI similar to that of Year 2. Interested readers may request the full calculator with relevant appendices 
from the authors at no charge. Table 1 provides a sample of the variables and returns that are possible from a simulator-based training program. 


Detailed below are the many factors to be considered in each cost category as well as limitations to the calculator. 


Investments 
The costs associated with adopting a simulator training program include: 


Cost of Simulator. The initial investment in purchasing the simulator device. The price point for many surgical simulators is in the neighborhood of $100,000 per de- 
vice. 

Annual Maintenance Fee. Most devices include one year of maintenance or service warranty with the purchase of the device. In following years, there is an annual 
fee to cover the installation of updates and repairs to the device. One must also allow for additional repairs beyond the normal maintenance. 

Facilities Costs. The equipment needs to be housed in a location that is conducive to sufficient access and training. A dedicated space is preferred, but some institu- 
tions use a shared space model to reduce this cost. This impacts the degree to which surgeons can use the devices on their own initiative which directly impacts train- 
ing efficiency. 

Staffing Costs. Most simulator devices require a knowledgeable trainer to instruct the surgeons on the use of the device and to facilitate data collection and the track- 
ing of performance improvement. This staff person does not necessarily have to be dedicated to the robotic simulators, but can be a shared resource that has addi- 
tional duties in the hospital system. The staffing cost can vary widely depending on the type of assignment used, such as hiring new personnel versus training an ex- 
isting employee. An additional consideration would be the loss of revenue from physicians, nurses, and other medical professionals during training sessions. 

Supplies & Materials. Because robotic simulators are electronic and use no real consumables like suture pads, needles, and fluids, the supplies and materials for con- 
tinuing to use them is minimal. Some typical supplies include: power strips, extension cords, cleaning wipes, spare parts, computer thumb drives, and computer 
adapters. 

The investment in simulator programs is typically highest at initial purchase for the equipment and much lower for each ensuing year. Therefore, the model separates 
the first year from years following. The expenses in years 2, 3, 4, etc. appear to be very similar. At this point there is little data on equipment fatigue with use. There is 
only published data on the RoSS device, for which master controllers and the pinch devices need to be replaced at 180 and 360 hours of use respectively °. We are 
therefore not able to identify the effective useful lifetime of a simulator device or a significant upturn in costs due to increased maintenance in later years. Insight into 
what to expect in robotics may be found in the operating expenses for laparoscopic training simulators which have been in use for many more years. 


Returns 


The returns that can be experienced through the implementation of a robotic simulator training program fall into four major categories: surgeon productivity, surgeon 
health, hospital costs, and alternative training costs. Each of these has many variables which contribute to financial returns or cost avoidance. 

Surgeon Productivity. The primary goal for most organizations creating a simulator training program is to increase the productivity of their surgeons. A model of sur- 
geon productivity begins with the number of surgeons who are trained and the amount by which this training can speed up a typical operation. Reducing the time re- 
quired to perform a surgical procedure can contribute to multiple variables. First, it may reduce staffing costs by allowing the staff that supports the surgeon to move 
on to other activities, or to work fewer paid hours during the day. In some cases, it may reduce total OR staffing due to the ability of the surgeon to control both the 
camera and the surgical arms concurrently, without the need for a human assistant. Second and most importantly, if the reduction in time is large enough, it may 
open a window in the OR schedule which is sufficient to perform an additional operation during the day. Third, if simulator-based training can reduce the variance in 
surgical times for procedures, it will make the scheduling of procedures more accurate. This would reduce the “slippage” that occurs in the daily schedule because a 
procedure takes twice as long as was scheduled. Smaller variances lead to more efficient scheduling which can reduce staffing and facilities costs. This model focuses 
on the impact of reducing surgical time sufficiently to open a window for an additional surgery. The impacts of variance across multiple instances of similar proce- 
dures is worthy of an entire dedicated study and is not included. 

Surgeon Health. Robotic systems significantly reduce the physical workload experienced by traditional open and MIS surgeons. All surgeons have a level of stamina 
that allows them to perform for a specific number of hours each day. Upon reaching this point, the muscles and mental focus of the surgeon are diminished by fa- 
tigue. Continuing to operate can bring risks both to the health of the patient and the surgeon. 

Rehearsal in a simulator has the effect of training the muscles which are used in surgery. Familiarization with the tasks also reduces the amount of mental and physi- 
cal effort that is required to complete each procedure. The extension of surgeon stamina and the reduction in energy expended can potentially lengthen the number 
of hours that a surgeon is able to perform optimally. 

This physical and mental training can extend the surgical day for a specific individual. It may also extend the length of the surgeon’s operating career by reducing re- 
petitive stress injuries and aging due to mental stress. The cumulative impact on the length of careers cannot be calculated at this point. Exploration of this effect is 
left for future researchers. 


The model of surgeon health included in this ROI model is limited to daily extensions of operating time. 


Hospital Costs. The model identifies four major hospital cost categories which may be reduced due to improved surgeon competence: instrument breakage, liability 
insurance, surgical error rate, and alternative training costs. 

Inexperienced surgeons can break the controls of the robot through the exertion of too much force and fighting against the electrical and mechanical components of 
the robot. When the surgeon damages the robot, the cost of many (but not all) repairs is covered by the maintenance agreement; but the significant cost to the hos- 
pital is in the disruption that occurs due to postponing or rescheduling an operation because the robot is no longer operable. Inexperienced surgeons can also dam- 
age the surgical instruments held by the robot. Each of these typically costs between $1,200 and $2,500. Frequent abuses come from forceful instrument collision and 
friction along the shafts. Simulator training can develop skill and dexterity with an instrument which can reduce this occurrence. 


As medicine becomes even more evidence based, there will be the opportunity to quantify the liabilities associated with specific surgeon competence levels. This may 
allow surgeons who can document additional training and acquired competence to reduce their medical liability costs when compared with surgeons who have less 
training. Experts whom we interviewed for this study indicated that they had already begun annual mandatory simulation-based exams as part of their internal risk 
management plan with the hope of realizing insurance savings in the future. 

The competence that comes from simulator-based training can reduce the number of errors that are made in robotic surgery. Some of these errors can require a re- 
turn to the operating room for a procedure which generates no additional revenue. 

There are multiple forms of training used to acquire, maintain, and extend the expertise of the surgeons. When a simulator program is introduced it can reduce the 
need for alternate forms of training. One popular form of robotic training today is through visitation and instruction in the OR itself. This long established and tradi- 
tional practice is used in all surgical specialties. But, it is known that “in OR” training results in an extension of the time to complete the procedure 11 Additional time 
is required to explain the procedure, the equipment, and the effects of actions to the trainee. Given the high cost of OR time, a reduction in OR-based live training can 
generate meaningful savings for a hospital by shortening the duration of these OR-based teaching sessions. 

There are two common forms of outsourced training. The first is bringing an instructor into the hospital system to teach a specific curriculum. The second is sending 
the surgeons to other facilities or congress meetings to acquire the skills they need. Both of these are subject to cost reductions when similar skills can be acquired 
and measured in the simulator program. Once equipped and experienced, the organization can also become a vendor of training services and simulator systems. Oth- 
er organizations may seek out such rental agreements to avoid their own capital purchasing costs. 


Two additional considerations include team training and surgical assist training. Some of the increased length of robotic surgical cases is due to set-up time and dock- 
ing times. Ensuring proper training of surgical assists and nurses in addition to physician training may decrease these times and costs. 


Limits 

There are upper limits to the improvements that can be achieved. For actual surgical procedures, these are driven by the number of robots available to perform sur- 
gery. Increasing surgeon availability beyond the capacity of the robotic OR will not provide improvements. An additional consideration is the availability of the robotic 
simulator for daily use or the training efficiency of the simulator device. If an institution chooses to invest in a DVSS trainer but does not have a dedicated training 
console to support it, then training availability will be limited when compared to those with a standalone trainer. 
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Table 1. RO! Calculator Sample Variables 





INVESTMENT Cost RETURNS 
(Year 1) 





Cost of Simulator $96,000 Surgeon Productivity Number of Surgeons Trained 





Annual Maintenance Fee $o Mean Length of Operation 


(Hours) 





Facility Costs $100,000 Time Saved per Procedure (Minutes) 10 





Staffing Costs $20,000 Length of Surgical Day 8 





Supplies & Materials $1,000 Revenue Per Surgery $20,000 


(Includes Admission) 











Surgical Days per Year 





Surgeon Health OR Stamina (Hours) 





Improved Stamina (Hours) 





Hospital Costs Liability Insurance $100,000 





Competence Discount 2% 





Average Instrument Breakage $5,000 





Breakage Reduction Factor 10% 





Surgical Error Rate 1% 





Error Improvement Rate 





Training Costs OR Training Time per Day (Hours) 





Number of In-house Courses 





External Training Event Cost (w/Travel) 





Number of External Events 





Training Reduction 














DISCUSSION 

In 1910, William Mayo famously stated in a graduation speech at Rush Medical College in Chicago that, “There is no excuse today for the surgeon to learn on the pa- 
tient.” In the ensuing decades, there have been many advances in the education of surgeons. Simulators offer the next improvement along this path with capabilities 
that are very difficult to match through any other mode of training. We are entering a period in which competence-based learning is becoming the standard. This is 

an environment in which simulators possess undeniable advantages and empower the preparation of demonstrably competent surgeons. 


Robotic simulators are available but are not yet a standardized part of the training process. As the airlines and the military discovered in training pilots and combat 
personnel, simulation devices provide so many advantages in the learning process that they have become a mandatory part of those training curricula. The expecta- 
tion is that the same will occur with robotic simulators over time. 

We recognize that many of the proposed financial and non-financial benefits are theoretical and that simulation in a well-designed curriculum purports more benefits 
than just the simulators themselves. There is an absence in the literature regarding the direct correlation between simulation and many of the positive impacts noted 
in this study. Despite this absence, there is evidence of three things that make these impacts real: the evidence of transfer of training with simulation as evidenced by 
validation studies, the presence of a steep learning curve in robotic surgery (trends of up to 100 cases before true competency is achieved), and the improvements 
that are seen in surgical outcomes with high volume surgeons and centers. We also know that hospital systems and practices are highly variable and all factors can- 
not be identified and/or quantified financially. If there is not a direct return on investment, we are not suggesting that simulation based education is of no value. We 
designed the calculator to capture simulation effects on a business practice not only to justify the investment but to identify best practices and ways to implement ro- 
botic surgical curriculum in a meaningful and comprehensive way. It is a tool to motivate discussions between both business and clinical leaders when making a pur- 
chasing decision for a simulator. 

To summarize, the use of simulators in training robotic surgeons requires an investment in equipment, staff, facilities, and supplies. But it also offers a return on this 
investment in surgeon productivity, surgeon health, hospital costs, and alternative training costs. The realizable returns will vary by institution based on the specifics 
of implementation and the existing ecosystem in which they are inserted. The ROI model in this report is one tool to assist in calculating the return that can be ex- 
pected by organizations that make the investment in these devices and training programs. 


Financial Disclosure: Florida Hospital received an unrestricted educational grant from Mimic Technologies Inc. to explore this concept. 
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Abstract 
Purpose Todetermine the impact of communication latency 
‘on telesurgical performance using the robotic simulator dV. 
“Trainer” 

Methods Surgeons were entolled during three robotic con- 
gresses. They were randomly assigned to a delay group 
(ranging from 100 to 1000ms). Each group performed three 
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times aset of four exercises on the simulator: the first attempt 
without delay (Base) and the last two attempts with delay 
(Warm-up and Test). The impact of different levels of latency 
was evaluated. 

Results. Thirty-seven surgeons were involved. The different 
latency groups achieved similar baseline performance witha 
‘mean task completion time of 207.2 s(p > 0.05). Inthe Test 
stage, the task duration increased gradually from 156.4 to 
310.7 as latency increased from 100 to $00ms. In separate 
‘groups, the task duration deteriorated from Base for latency 
stages at delays >300ms, and the errors increased at $00ms 
and above (jp <0.05). The subjects’ performance tended to 
improve from the Warm-up to the Test period. Few subjects 
‘completed the tasks with a delay higher than 700 ms. 
Conclusion Gradually increasing latency has a growing 
impact on performances. Measurable deterioration of per 
formance begins at 300ms, Delays higher than 700ms are 
difficult to manage especially in more complex tasks, Sur 
‘geons showed the potential to adapt to delay and may be 
‘wained to improve their telesurgical performance at lower: 
latency levels, 








Keywords  Telesurgery - Telemedicine/methods - 
‘Computer simulation - Robotic simulator Internet 


Abbreviations 
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ms Millisecond 
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CT2 Camera Targeting 2 

TRI ‘Thread the Ring 1 

EDI Energy Dissection 1 
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Introduction 


Robotic surgery was noted to bein its infancy in 2004, [1] but 
now this advanced technology is on its way to young adult 
hhood [2]. It has become a standard in complex surgery [3]. 
‘The mature experience will likely include the achievement 
‘of remote telesurgery a future challenge for robotic surgeons 
1451. 

The first transatlantic human telesurgery procedure was, 
performed in 2001 [6]. Since the proof of concept, elesurgery 
remains a complex and uncommon process that holds 
promise in overcoming challenging situations (remote medi 
cine for underserved regions, surgery in the battlefield, 
surgery in space, ete.) [7.8]. Many teams have worked on the 
telesurgery process and tried to achieve remote telesurgery 
procedures using available technical resources forthe video 
flux transfer (7,9, 10]. In telesurgery, the control signal sent 
from the master console is transferred over a network to the 
robot arms followed by a corresponding movement of the 
surgical instruments, The video images are then retuned to 
the surgeon site. The data transmission requires an encoding, 
transmission, and decoding process in which a time delay, or 
latency, is inevitably produced. Latency is correlated with 
the amount of data and the quality of network. The first 
transatlantic human telesurgery (with the Zeus robot) used 
sophisticated dedicated asynchronous transfer mode (ATM) 
lines witha transmission delay around 150ms [6]. Dedicated 
lines, however, are not always feasible in routine clinical sit 
uations, The public Internet bridging the world could be an 
easy and accessible resource to transmit this data. Even so, 
the network availability would be atthe price of increasing 
latency measured approximately 450-900ms [11] 

It would be valuable to clarify the impact of the latency 
‘on surgical performances before future implementations of 
telesurgery. Two thresholds need to be established: The firs 
isthe smallest latency that canbe detected by surgeons which 
will influence their performance, and the second is the level 
of latency that makes the surgery unsafe. Unsafe surgery is 
associated with an increase in errors. A previous study on 
this topic highlighted the impact of delay on performance 
degradation using the dV-Trainer®. The authors evaluated 
the effects of delay varying between 100 and 1000ms, and 
found that latencies <300ms had a small impact on per- 
formance. Subjective evaluation then suggested that surgery 
became quite difficult at delays >800 ms [12], However, this 
study only included medical students as the subjects. Addi 
tional experiments should be performed with experienced 
surgeons, especially those experienced with robotic systems 
which would be needed to implement telesurgical proce- 
dures. 

The present study aims to evaluate, on a surgeon popula- 
tion, the impact of different latency levels on performances 
in four simulated robotic tasks. 
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‘Material and methods 
Exercises and subjects, 


We designed a prospective, observational study conducted 
fn the robotic surgical simulator €V-Trainee® (Mimic tech 
nologies Inc, Seatle, USA). This tool has demonstrated 
face, content. construct, and concurrent validity in previ- 
fos studies [13,14], Based on expert opinion and literature 
review [14,15]. we chose four exercises for the test that 
would be performed in a constant easy-to-ificult order: 
(a) Peg-Board | (PB1)—pick up and transfer rings sequen: 
tially from the Peg-Board toa single peg on the for; (b) 
(Camera Targeting 2(CT2)—manipulae the camera to pre 
cisely focus and 200m on a target sphere: pick up and move 
4 stone into a designated basket: (c) Thread the Rings 1 
(TRI)—pass a needle and suture through a number of flex- 
ible eyelets; (d) Energy Dissection | (ED1)—isolate a large 
blood vessel by cauterizing and cutting small branching 
blood vessels that anchor the lage vessel (Fig. 1). Both 
basic (endowtist manipulation, camera control, clutching) 
and challenging (suturing, dissection) skills were covered 
/ simulator permitted 





with these exercises, The 4V-Trainer® 
us to introduce fixed latencies into the exercises between 
the gesture on the grips and the visual feedback on the con- 
sole. 

Afier institutional review board approval, we recruited 
subjects—fellows and attending surgeons—during three 
robotic surgery conferences. All the experiments involving 
‘human participants were in accordance with the ethical stan 
dards of the institutional research committee, as well as the 
1964 Helsinki Declaration and its later amendments or com- 
parable ethical standards. Informed consent was obtained 
from all individual participants. 


Procedures 


Each participant received a unique identification number 
under Which all his/her data would be collected, and then 
completed a questionnaire concerning demographic data 
(including surgical experience and related activities). 

Each subject was randomly and blindly assigned a 
latency varying between 100 and 1000ms with inere 
ments of 100ms. Before the tials on dV-Trainer®, they 
received standard instruction on its use in a familiariza- 
tion period. After that, they performed all four exercises 
in order without delay (Base). The results provided their 
baseline performance. Then they repeated the same set of 
exercises twice with the assigned latency (Warm-up and 
‘Test). The Warm-up period allowed them to become famil- 
iar with latency and to acquire short-term adaptation (Fig 
2» 
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Fig, 1 The four dV-Trainee® 
exercises: Peg-Board (a), 
‘Camera Targeting 2 (b), Thread 
the Ring 1 (e), and Enessy 
Dissection 1 (@) 


Peg Board 1 


‘Assignment ofa da 





(varying batween 700 and 7,000 ms) 





“Standard instruction on tho use of AV-Traner 


Base: one attempt of PY, C72, TRY 





(EDI without delay 


(Warm 





| Test: one attempt of he four exercises with the assigned delay 


Fig, 2 Experimental procedures 


Metries 


‘The dV-trainer includes a built-in scoring system. The val- 
ues of the following metrics were automatically recorded 
after each exercise: time to complete the exercise (in sec 
‘onds), instrument motion (in centimeters), master workspace 
‘ange (in centimeters), excessive instrument force (in sec- 
fonds), instruments out of view (in centimeters), instru: 
ment collisions, drops, etc. An overall. score representing 
‘a combination of these criteria was also automatically 
generated, 

Based on our experience, the task completion time is the 
‘most sensitive and reliable measure to the impact of delay 
[12]. We thus chose this measure to represent the results. In 
addition, the mean score of all error metrics was calculated 
in order to evaluate the latency impact on errors. 


Thread the Ring 1 






Camera Targeting 2 








Energy Dissection 1 


Statistics 


Data were analyzed using the R statistical software. A 
rtepeated-measures ANOVA (mixed-effects model) was used 
to determine the differences in performances between Vari 
‘ous latency groups (with FDR p value correction), and also 
between the three periods in each latency group (with Holm 
correction). Statistical significance was determined at p< 
0.05. 


Results 
Complete data 


Final data were derived from 37 surgeons. Twenty-three per 
sons had robotic experience, with an average of 2.7 years 
(ranging from 1 to 9years). All subjects completed the three 
sages from Base to Test, but some of them did not complete 
all the exercises. For example, four subjects were included 
in the 100ms group, but one of them did not complete the 
exercises of CT2 and TRI. The groups from 700 to 1000 ms 
were combined due to the limited subject number (Table 1), 


Results actoss exercises 


‘The different latency groups achieved similar baseline per: 
formance with a mean task completion time of 207.28 (p > 
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0.05), An increasing tendency of the task duration withdelay os 
was observed in the two latency stages. In the Test period, 
the mean task duration increased from 156.4 at 100ms to 
310.7s at $00ms. When comparing this measure between 
any two latency groups, statistical significance was achieved» 
in the comparisons of the 100ms group versus the 400 and 
500ms groups (p < 0.05: Fig. 3). ms 

Subjects demonstrated the tendency to improve their per 
formances from the Warm-up to the Test period. The task 
‘completion time deteriorated from the baseline to the two at 
latency stages at 300ms andabove, although statistical signif- 
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7 3 cance was not achieved at 300 ms due to the limited subject 1 
i number (Fig. 3). The comparison results between the three 
rectnd periods in each Ilency group are Mustrated in Fg. 4 - 
‘The mean erorscore deteriorated from baseline ta latency sw 
stages at $00ms and above (p< 0.05). For example, ino» 
z i 
# 
Geel 
fisi, 
weanvdebed ca a a 
2 Latancy (ns 
z Fig. 3 Toe mean tsk completion time aco the fou test exerises 
2 in each Inteney group. * Difference was determined compared to the 
a Wms group (p< 0.05). The groups of 611) and 700-1000ms were 
26 otneluded due to insufficient data in certain exercises 
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Fig. 5 Comparisons of the mean error score between the thre periods 
in each latency group ("p< 0.05; **p -< (101). The group 6iXIms 
includes only the resulls across PBL, CT2, and EDI; the group 700 
TIkNIm eludes the results aeross PBL and CT? 


500ms group, the score decreased from 168.2 (out of 200) 
to 138.5 from the Base to the Test period (Fig. 5) 


Results in separate exercises 


‘An increasing tendency of the task completion time with 
latency was observed in the two latency periods of the four 
exercises. The degradation of performances between baseline 


and latency stages started with 300, S00, 100, and 300ms in 
PBI, CT2, TRI, and EDI, respectively (Fig. 6). 


Incomplete data 





Eighty incomplete exercises in latency stages derived from 
26 subjects were identified. They included 18 PB1, 18 CT2, 
26 TRI. and 18 EDI. Subjects were physically unable to 
complete these delayed exercises. Fifty-three (66.25 %) exer: 
cises were stopped by the subjects at a mean time of 9.8 min 
(586.01 + 14,545). The ratio of incomplete exercises was 
relatively higher in high-delay groups (Fig. 7). 














Discussion 


We aimed to determine the latency effects on surgical per 
formances in experienced surgeons who are unfamiliar with 
latency and the simulator device, to establish the thresh: 
‘old delays in telesurgery. Overall, the gradually increasing 
latency has an increasing impact on performances, and 
the performance deterioration consistently begins at 300ms. 
Latencies of 100 and 200ms seemed to have no clear effect, 
and the 100ms group had improving performance from the 
Base to the Test stage. This improvement likely corresponds 
10 the leaming effects of basic simulator manipulation and 
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Fig. 6 ‘The mean task completion time in each Itency group of the four exerises: Peg-Boar | (a), Camera Targeting 2 (b), Thread the Ring 1 
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Fig, 7 Thenumbers of complete and identified incomplete exercise at 
cach latency level, The aambers above the hars represent the percentage 
‘of incomplete exercises 


further proves that 100ms does not have a significant influ 
cence. For the superior threshold, delays equal or higher than 
700ms seem to be difficult to manage especially in com- 
plex tasks. Only one subject was able to complete the tasks 
at 700ms, and only the easiest exercises (PB1, CT2) were 
finished at 800-1000 m. In the previous study with trained 
‘medical students, the similar threshold was highlighted and 
the authors suggested telementoring as a safer choice [12] 
‘Telementoring is an application of telemedicine that involves 
the remote guidance of a procedure when the local operator 
thas limited experience with the technique [16]. However, in 
this study, the error rate significantly increased from non= 
latency to latency stages at delays >500ms, which may 
indicate an increase in surgical risk. We would consider this 
value as the superior threshold, and telesurgery should not 
be recommended in this condition for most surgeons [17] 
‘This does not mean that procedures cannot be performed at 
hhigher-latency levels, and results could be better for experi 
enced robotic surgeons, especially when given an opportunity 
to rehearse in an environment including lateney, such as with 
a simulator, Current research is still limited, and outcome 
data are lacking to demonstrate the feasibility and safety of 
telesurgery with high delays. In a previous published study. 
‘a nephrectomy was performed on a swine under a delay of 
900ms. Two surgeons performed the procedure, one in the 
remote site console and the other in the local site console 
[11 In this article, no outcome data Were provided, such as 
surgical performance and the mental stress of surgeons. 
‘Surgeons have been shown to have the potentials to-adapt 
to delays [18]. Similar tendency was also observed in our 
study: Performances improved from Warm-up to Test. It sug- 
gests that surgeons may be trained on latency to improve 
their telesurgical performance. However, the improvement 
‘observed here is not clearly attributable to adaptation through 
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experience with latency. It may also be the result of improve- 
‘ments in psychomotor simulator manipulation. Despite the 
‘overall tendency across exercises, our results also demon- 
strate that the impact of latency is related to the difficulty 
of procedures, Latency affected performances on different 
levels forthe four chosen exercises: The performance deteri 
‘oration started ata high delay (300 ms) forthe simple exercise 
CT? and at a low level (100ms) in the more challenging 
‘TRI, This fact indicates that the minimum influential and 
the maximum acceptable delays could be different in surgi 
cal procedures with different complexity 

For the challenging exercises that may better represent real 
surgical scenarios, we have chosen TRI and EDI instead of 
the more complex exercises like "Suture Sponge” or “Tubes.” 
‘This is because many surgeons were not sufliciently familiar 
(or proficient) with the robot or the simulator. In this study, 
fw tasks were completed at delays higher than 700 ms, One 
‘might anticipate that the results would be even worse ifapply. 
ing more challenging exercises. 

Participants have demonstrated the efforts to complete the 
tasks even with considerable latencies. In the identified 80 
incomplete exercises, only a few subjects terminated their 
participation soon after beginning. The mean duration of 
‘attempt was 7.5 min per exercise. This effort could minimize 
the bias of experiments. It is also interesting to observe that 
many persons stopped at about 10min. It seems that this is 
‘a threshold beyond which surgeons could no longer endure 
the effects of latency. 

‘This study has potential limitations: Although we recruited 
more than 60 surgeons, the final completion rate was lower 
than expected. The small number of subjects in each latency 
‘group isa shortcoming of the study. We did not merge differ: 
tent latency groups because the objective was to evaluate the 
impact of each latency level, and an interval of 100ms may 
already cause difference, Also the distribution of subjects was 
not equivalent in different latency groups, primarily due to 
subjects choosing to terminate their participation before com: 
pleting the entire experiment, Fewer subjects were included 
in the 300ms group. Moreover, many surgeons failed to 
complete the tasks at high delays due to the difficulty of 
‘manipulation under these conditions. In addition, all subjects 
were novices in telesurgery (or latencies) since this technol- 
‘ogy is currently only available in research settings. 

‘A complementary study will be necessary to assess the 
performance degradation induced by latency on robotic 
surgery experts, and to investigate whether latency training 
could be used to overcome the challenges of telesurgery. 


Conclusion 


‘This study was conducted on surgeons with limited experi 
cence using the dV-Trainer simulator, and the results demon- 
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strated that performances (time to perform, score, error) 
deteriorate gradually as latency increases. ‘The impact of 
delay is related to the difficulty of the procedures, but over- 
all, delays of 100 to 200ms have no significant impact, and a 
delay higher than 500 ms causes a noticeable increase in sur- 
‘sical risk. Surgery becomes extremely difficult and should be 
avoided at delays higher than 700ms, Telementoring could 
be an option in this situation, Surgeons have the potential to 
adapt to latency, and they may be trained to improve their 
telesurgical performances using devices like simulators of 
robotic systems. 
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R obotic ie 


intuitive Mimic dV-Trainer 





e Simulators 











Features [Dvss 


faV-Trainer 


Ross 


[Robotix Mentor 





[System Manufacturer | Intuitive Surgical Inc 


[Mimic Technologies Ine. 


Simulated Surgical Systems LLC 


Simbionix Lid, 





[Specifications Depth 41” Height 65" Width 
(Complete System) | 40° 120 or 240V power 


Depih 36° Height 59° Width 54° 120 
lor 240V power 


Depth 44" Height 77" Widih 45° 120) 
Jor 240V power 


Depth 36° Height 48° Width 49° 120 
Jor 240V power 





Visual Resolution | VGA 10045 768 


VGA 1004x768 


VGA Gi0x 380 


WGA 10a4 x 168 





‘Components [Customized computer attached 
to da Vinei surgical console 


[Standard computer, visual System 
[with hand controls, foot pedals 


Single integrated custom simulation 
device 


Standard computer, visual system 
with hand controls, foot pedals 





[Support Equipment | da Vinci Si surgical console 


[Adjustable table, touch sereen 
monitor, keyboard, mouse, protective 
Jcover, custom shipping container 


TSE adapter, Keyboard, mouse 


None 





Exercises 35 simulation exercises (30 by 
Mimic, 5 by Simbionix) 


(63 simulation exercises 


352 simulation exercises 


50 simulation exercises 





[Optional Software PC-based Simulation 
management 


[Mshare curriculum sharing web site 


Video and Haptics-based Procedure 
Exercises (HST) 


Uro, Gyn Procedural Modules 





[Scoring Method [Scaled 0-100% with passing 
thresholds in multiple skill areas 


[Proficiency-based metric and point 
system with passing thresholds in 
multiple skill areas 


Point system with passing thresholds 
in multiple skill areas. 


Proficiency-based metric and point 
system with passing thresholds in 
multiple skill areas 





[Student Data [Custom control application for 
Management external PC. Export via USB 
memory stick. 


[Export student data to delimited data 
file and graphical reports, 


Export student data to delimited data 
file, 


[Export student data to delimited data 
fle and graphical reports 





Curriculum None 
Customization 


[Select any combination of exercises. 
Set passing thresholds and 
conditions. 


Select specifically grouped exercises, 
Set passing thresholds. 


(Online curriculum development 





(Administrator (Greate student accounts on 
[Functions external PC. Import via USB 
memory stick, 


[Create student accounts, Customize 


[Create student accounts, Customize 
curriculum, 


[Create student accounts, export data, 
create curriculum, 





[System Setup [None 


[Calibrate hand controls 


[Calibrate hand controls 


(Calibrate goggles 





[System Security [Student account ID and 
password. 


[Administrator password, Student 
[account ID and password, Guest 
account. 


PC password, Administrator 
password, Student account ID and 
password, 


Administrator password, Student 
account ID and password, Guest 
account 





[Simulator Base Price [585,000 


[599,200 


$126,000 


875,000 





[Support Equip Price _ [5500.00 


$9,800 


30 


0 








Total Functional Price [9585,000 


Smith, Truong, Perez 2015 





[S109,000 





3126 000 





875,000 





Types of Validation 
McDougall 2007 


Dvss 


dV-Trainer 


RoSS 





Face 
Subjective realism of the simulator 


Hung 2011 
Kelly 2012 
Liss 2012 
Tanaka 2014 


Lendvay 2008 
Kenney 2009 
Sethi 2009 
Perrenot 2011 
Korets, 2011 
Lee 2012 
Tanaka 2014 


Seixas-Mikelus 2010 
Stegemann, 2012 
Tanaka 2014 





Content 
Judgment of appropriateness as a 
teaching modality 


Hung 2011 
Kelly 2012 
Liss 2012 
Tanaka 2014 


Kenney 2009 
Sethi 2009 
Perrenot 2011 
Lee 2012 
Tanaka 2014 


Seixas-Mikelus 2010 
Colaco, 2012 
Tanaka 2014 





Construct 
Able to distinguish experienced 
from inexperienced surgeon 


Hung 2011 
Kelly 2012 

Liss 2012 
Finnegan 2012 
Tanaka 2014 


Kenney 2009 
Korets, 2011 
Perrenot 2011 
Lee 2012 
Tanaka 2014 


Raza, 2013 
Tanaka 2014 





Concurrent 
Extent to which simulator correlates 
with “gold standard” 


Hung 2012 


Lerner 2010 
Perrenot 2011 
Korets 2011 
Lee 2012 


Chowriappa, 2013 





Predictive 
Extent to which simulator predicts 
future performance 

Smith, Truong, Perez 2015 





Hung 2012 
Culligan 2014 











Core Skills 


Content 


Simulators 





dV-Trainer 


RoSS 


Dvss 


Robotix 
Mentor 





Wristed Bimanual 
Manipulation 
Maneuver Camera 


Master Cntrl Clutching 


Use 3rd Arm 


Depth/3D Perception 


Aware Instrument 
Force 


Liu & Curet, 2015 


Tasks 
Metrics 


Tasks 
Metrics 


Tasks 
Metrics 


Tasks 
Metrics 


Tasks 
Metrics 


Tasks 
Metrics 





Yes 
No 
No 


No 
M Wrk Rng 


Yes 
No 


No 
No 


No 
Ex Inst Fre 





Yes 
No 


Yes 
No 


Yes 
No 





Yes 


M Wrk Rng 


Ex Inst Fre 





Yes 
No 





Nature of Validation in Surgical Simulation 





“Types of Validation” “Sources of Evidence” 


AERA/APA, 1974 AERA/APA, 1990 
McDougall, 2007 Korndorffer, 2010 
Face Content 
Content Process Response 
Construct Internal Structure 
Concurrent Relations to Other Variables 
Predictive Consequences 


As of 2010, 100% of the studies in Surgical education community is 
the published literature have used still struggling to understand how 
this model of validation. to apply this model. Itis not in 
general use at this time. 





1, Robot. Accessibility given cost of device (~$1.6 million) requires 
support from manufacturer. 


2. Simulator. Minimal device cost, operational support costs (staff and 
facilities) are difficult. 


3. Curriculum. Authoritative, standardized, objective. 
4. Metrics. Validated, applicable across devices and curriculum. 
5. Certification. Enforced by boards, societies, and hospitals. 


Fundamentals of Robotic Surgery 
addresses #3, 4, and 5. 





‘Smith R, Truong M, Perez M. (an 2015), Comparative analysis of he functionality of simulators ofthe da Vinci surgical robot. Surg Endosc, 29:972-983, 
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AERA/APA, (1990 & 1974). The Standards for educational and psychological testing. htp:/www.apa.org/science/programsitestingistandards.aspx 

Komdorffer R, Kasten 5) , Downing SM. (2010).A call forthe utilization of consensus standards in the surgical education literature, Amer Surg, 199, 99-104, 

Hung AJ, Zehnder P, Patil MB. (2011), Face, contentand construct validity of a novel robotic surgery simulator.) Urology, 186(3):1019-24, 

Kelly DC, Margules AC, Kundavaram CR. (2012).Face, content, and construct validation ofthe da Vinci Skills Simulator.) Urology, 79(5):1068-72. 

Liss MA, Abdelshehid C, Quach S. (2012). Validation, Correlation, and Comparison ofthe da Vinci Trainen™ ) and the da Vinci Surgical Skills Simulator(™) Using the Mi 
Software for Urologic Robotic Surgical Education. Endourol, 26(12):1629-34, 
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Kenney P, Wszolek M, Gould J, Libertino J , Moinzadeh A. (2008). Face, Content, and Construct Validity of dV-Trainer, a Novel Virtual Reality Simulator for Robotic Surgery. 
Urology, 73(6):1288-2. 

Sethi AS, Peine W), Mohammadi Y. (2008) Validation of a novel virtual realty robotic simulator.) Endourol, 23( 
Perrenot C, Perez M, Tran N. (2012) The virtual reality simulator dV-Trainer(6) is a valid assessmenttool for robotic surgical skis.) Surg Endosc, 26(9):2587-93, 
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The Role of Simulation in Training 


Domain of Skills 


Be go 
ar" 
A 





Pe WETALE Ve (253) Disadvantages 

Cost of device Accuracy of replication 
Training support costs Completeness of replication 
Programmable curriculum Additional system support 
Automated metrics Lag device upgrades 
Progression of skills Community acceptance 


* Obvious: Training in failure modes and recovery, safe set- 
up and verification, surgeon skills, team roles. 

* Forms: training with equipment, dry labs, tissue, role 
playing, simulated emergency. 

* Issues: Cost of equipment, staff, and services. Training ROI 
is not good enough to motivate any 3” party to offer 
training. Financially only makes sense for device 
manufacturer. 


Create standard materials 

Establish benchmarks 

Require test & evaluation 

Issue: How do societies/hospitals develop expertise to 
establish these in the early phases? Bootstrap needed to 
get started. Need trusted partnership with device 
companies. 


* Cross correlate multiple training methods & curricula. Use 
2-3 different methods to measure the same skill. Matching 
errors in each method indicate a real lack of skill, notjust 
an isolated mistake (Polya, 1945). 

¢ Many more tricks in the mathematic and psychometric 
toolkit. 


Dry lab tools, curriculum & standards 
Cadaver and animal labs 
VR/3D sims for skills and procedures 
- Early wireframe approaches used by NASA and DOD 
Team role playing and curriculum tools 
OR staff training 
Emergency procedures course 


Robotic training with porcine models induces less workload than virtual 
reality robotic simulators for urology resident trainees 
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INTRODUCTION 


Robotic assisted urologic surgery is predicted to continue to grow in usage in the coming 
years, and residents trained in urology will increasingly be expected to be proficient in robotic 
surgery’. The complexity of robotic technology, its steep learning curve, and work-hour 
limitation of resident trainees make incorporating robotic training into residency a challenging 
task. Experts suggest that learning as a bedside assistant for robotic surgery has a rapid plateau; 
many programs are now utilizing physician's assistants and surgical technicians for bedside 
duties in order to free the residents for console training”. In high volume programs it remains 
difficult for residents to gain hands-on console time due to their insufficient skill set and the 
complexity of most procedures 

Robotic simulation training tools can therefore be utilized by novice trainees to shorten 
the learning curve and improve operative skills in a low-risk environment. In pursuit of 
mproving the quality of residents’ education, the Southeastern Section of the American 
Urological Association (SES AUA) hosts an annual robotic training course for its residents. The 
aim of this study is to evaluate robotic simulation workload and stress levels on urology resident 
trainees utilizing porcine models and virtual reality robotic simulators during this workshop. 











Material and Methods. 


Select residents from each of the 14 training programs of SES AUA are invited to 
Orlando, FL, for a 2-day robotics training course. Up to3 residents were invited from each 
training program. The 2015 cohort of residents represented a wider range of training and 
diversity in experience than in previous courses being exposed to robotic surgery early at their 
‘home institutions. Volunteer faculty were recruited from multiple SES AUA training programs. 

SES AUA resident assessment of the workload associated with robotic simulation and 
live robotic surgery in a porcine model were assessed using the NASA Task Load Index (NASA- 
TLX). The NASA-TLX assesses workload along six dimensions: mental demand, physical 
demand, temporal demand, performance, effort, and frustration’, Each is measured on a 21-point 
scale between "Very Low" and "Very High" (Appendix 2). 








‘The SES outlined below’, 





SAUA robotics course undertaken by the residents 





RoBoric Course Day I 

A full didactic session broken into 3 components. Component | covered the basics of 
robotic surgery including room set-up, bedside assistance, and console essentials. Component 2 
covered several aspects of robotic kidney surgery including patient positioning, port placement, 
and surgical techniques. Component 3 focused on robotic prostate surgery including port 
placement and different surgical techniques. Didactics were supplemented with surgical videos 
and discussions of difficult surgical scenarios and possible complications. 


ROBOTIC CouRSE Day 2 

‘The trainees were divided into two groups. Half were asked to perform skill tasks on the 
Mimic da Vinci-Trainer (MdVT, Mimic Technologies, Inc., Seattle, WA, U.S.A) for four hours 
while the other half performed set tasks in a live nephrectomy on porcine model using the da 
Vinci Xi robot (Intuitive Surgical Inc., Sunnyvale, CA). After the four hours the groups changed 
places and continued for another 4-hour ses 





ion. 


SIMULATION SECTION 
In the 4-hour MdVT simulation session, trainees were first given a tutorial of the console 
and its functionality. The trainees then proceeded to complete 5 applications of increasing 
difficulty and required skills. The first application, “pick and place”, involved simple movements 
of pyramidal jacks into corresponding colored bowls and is used to orient the trainee to the 
simulator. The second application, “peg board” is more advanced and required the trainee to 
clutch hand instruments while moving the camera, which involves coordinated hand and foot 
movements. The third application, “ring walk”, involved moving a ring over a curved bar 
without touching the bar with any portion of the ring. This drill requires all the above skills as 
well as maintaining awareness and accuracy with the ring position in three dimensions. The 
fourth application, “thread the rings”, involves passing a curved needle through rings positioned 
at different angles without touching the ring with any part of the needle. This drill teaches 
trainees good suturing technique. The last application, “tubes 2”, is the most challenging and 
realistic. This drill is designed to replicate the performance of an urethrovesical anastomosis. It 
utilizes all of the above skills including accuracy, coordination, and sufficient needle control, 


ANIMAL TRAINING SECTION 

In the 4-hour porcine model live surgery session, all trainees spent one hour performing 
cystostomies and cystorrhaphies. They then spent thirty minutes practicing port insertion and 
robot docking. Finally, for 2.5 hours, trainees conducted a bilateral nephrectomy which included 
artery, vein, kidney and ureter dissections and ligation. 








QUESTIONNAIRE 

All trainees were asked to complete a I-page demographic questionnaire following the 
MAVT session (Appendix 1). They were also asked to complete the NASA TLX I-page 
questionnaire following both the MdVT simulation and live animal model sessions (Appendix 
2). 








STATISTICAL ANALYSIS 
Descriptive (range, mean, standard deviation, frequency). t-tests and analysis of variance 
using SAS (version 9.2; SAS Institute Inc., Cary, NC, U.S.A.). 


RESULTS 

‘Twenty-one residents from 14 programs in the SES AUA participated in this course. 
Seventeen (80.9%) had used a robotic console during an actual surgical case, while four did not. 
‘The distribution of the different levels of training among the residents is shown in Figure 1. 
Unlike previous years’ courses when only senior or chief residents participated, this course 
included more junior residents. This reflects a shift toward early exposure to robotic surgery 
during urology training in most academic programs. The number of robotic or laparoscopic 
surgeries performed or assisted in by residents at different levels of training is shown in Figure 2. 
‘Trainees’ satisfaction with their program robotic surgery training was assessed (Figure 3). Of the 
17 residents who performed actual robotic surgery, 7 (41.2%) stated that the simulator replicates 
real-life robotic surgery, while 10 (58.8%) stated that it did not. 





‘What year urology resident are you?” 


‘Number of Residents 
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Figure 1, Robotic Simulator Questionnaire: Question | results 
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Figure 2, Robotic Simulator Questionnaire: Question 4 results 


How do you rate your robotic training during residency? 


aliad 


PGU-3 
Level of Training 


[Number of Residents 


‘Poor Fair "Average Excellent 
Figure 3. Robotic Simulator Questionnaire: Question 5 results 


The NASA-TLX scores were converted to a 0-100 scale with 5 point increments. The 
raw TLX method was employed to eliminate the weight variability of the different TLX scales. 
To assess the NASA-TLX data at two interfaces (simulator vs. animal model) for the different 
levels of training (year of residency), a 4 x 2 x 6 (residency, interface, and TLX scale, 
respectively) mixed ANOVA was computed. The Greenhouse-Geisser correction was used to 
correct for the sphericity assumption, The ANOVA indicated a significant main effect for TLX 





scales, F (3.91, 66.44) = 4.93, p = 0.002, nhpariai = 0.225, as well as a significant interface by TLX 
scales interaction, F (3.73, 63.42) P = 0.016, Maria = 0.166). None of the other main 
effects and interactions were significant. To further analyze the TLX main effects, Bonferroni- 
corrected repeated-measures t-tests were computed to determine which TLX scales differed 
significantly from each other; type-I error rate per comparison was set to 0.003. Means of the 
TLX scales are presented in Figure 4. As can be seen in Figure 4, effort resulted in the highest 
score. The Bonferroni corrected t-tests indicated that mental demand was significantly higher 
than physical demand (r (20) = 4.05, p = 0.001) and then frustration (t (20) = 3.52, p = 0.002). 
Further, temporal demand was significantly higher than physical demand (t (20) = 2.90, p = 
0.009) and that effort was significantly higher than physical demand (1 (20) = 6.52, p <0.001)), 
temporal demand (f (20) = 5.12, p < 0.001), performance (t (20) = 5.15, p < 0.001), and 
frustration (t (20) = 6.90, p < 0.001). 
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Figure 4. Mean scores of the NASA-TLX scales. Note: Error bars refer to standard error of the 
mean, 


The analysis of the interface by TLX interaction was further analyzed to determine 
whether the scores of each of the six TLX scales varied across the two interfaces. On that end, 
Bonferroni-corrected repeated-measures t-tests were computed; type-I error rate per comparison 
‘was set at alpha = 0.008. The means of the TLX scores observed at the two interfaces are in 
Figure 5. The only significance was observed for frustration, which was significantly higher at 
the simulation than the animal model, 1(20) = 4.12, p = 0.001). 
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Figure 5. Mean scores of the NASA-TLX scales in simulation versus animal model 


Discussion 


Robotic surgery is increasing in popularity in the field of urology due to its minimal 
invasiveness, reduced risk of complications and shortened hospital stay. This growing trend is 
evident in our results, The majority of the trainees in the 2015 course (80%) reported live 
console exposure. In contrast with a similar survey conducted in 2013 in a group of SES AUA 
trainees, only 56.9% of the trainees that year reported having had robotic console time’, During 
the 2014 annual training course 92% of the trainees reported performing live robotic surgery at 
their home institution’, Despite these increasing numbers, there is a lack of standardization and 
certification process for urology residents in robotic surgery. Furthermore, there is no 
standardized training protocol for residents learning robotic surgery across the various training 
programs, Gover et al, suggested a threshold of 25-30 cases for a novice surgeon to begin to 
operate the foot pedals and controls safely and intuitively’, Only 4 (19%) of our trainees reported 
having performed more than 25 cases. 

Robotic surgery simulators have been proposed to narrow the gap of novice trainees’ skill 
levels’. They would also help establish the basics of important operative skills such as eye-hand- 
foot coordination and comfort using the console controls and foot pedals. Our program chose to 
use the Mimic dV-Trainer simulator for training, which is one of the most established virtual 
robotic surgical simulators today. The current version of the dV-Trainer (version 2) contains 51 
exercises organized into nine categories’. This device also includes video and audio instructions 
on how to use the robotic equipment, moving through progressively more difficult skills. Kenney 
et al. suggested a content and construct validity in their study*. There are other robotic simulators 
that are commercially available such as the da Vinci Skills Simulator (DVSS) (Intuitive Surgical 
Inc., Sunnyvale, CA, U.S.A.), the Robotic Surgical Simulator (RoSS; Simulated Surgical 
Systems, Buffalo, NY, U.S.A.) and Robotix Mentor™ (3D Systems, Simbionix Products, 
Cleveland, OH, U.S.A.). However, they do not meet yet the same degree of validity as MdVT. 











ly, Lemer et al, demonstrated that residents who trained on the MdVT 
outperformed those who trained solely on the real da Vinci surgical system (Intuitive Surgical 
Inc., Sunnyvale, CA) when taking a robotic skills assessment using the real da Vinci system’. 

However, even MdVT simulator usage is not without its limitations. The MdVT costs 
from $85,000 to $100,000 to purchase with additional annual maintenance fees. Additionally 
there are no surgical procedure simulation drills, only specific skill tasks like the ones uses in 
this group. There are currently no urologic-specific procedure modules or simulation drills 
available, only general surgical skill tasks like the ones used in this group. This limitation could 
limit the pace of learning and might not translate to better operative skills without supplemental 
live surgery console time. Therefore, work on more realistic 3D case simulations to advance 
clinical decision-making and procedural knowledge is currently in progress. The animal model 
used in this analysis costs roughly $500/nr"”. It also lacks realistic human anatomy and might 
provide a false sense of security which could increase risks to future patients''. Future work 
should focus on developing urology-specific training modules such as radical prostatectomy and 
partial nephrectomy simulations. The existing application only hones skills used in general 
robotic surgery and is not necessarily reflective of skills needed to perform urologic robotic 
surgery. 

Educators and companies have yet to determine the best model to use for teaching robotic 
surgery. Many factors must be taken into consideration including the cost, availability of expert 
faculty, legal responsibility of supervising faculty, risk to patients, and the additional workload 
on trainees. This analysis has been conducted with aim to assess the latter factor. Trainees agreed 
that effort exceeded the other five workload dimensions asked on the NASA-TLX. Trainees felt 
that they worked physically and mentally hard to accomplish their tasks both on the simulator 
and the live animal model. In general, the performance level was the same for both parts of 
training which suggests that the training accomplished its goal 

Mental demand greatly surpassed physical demand, as expected. The trainees also 
reported being less frustrated with the live animal model than with the simulator. This could be 
due to the trainees’ familiarity with live anatomical structures over skill set simulations which 
remains a real challenge to novice surgeons. Simulators also provide metrics to score specific 
performance traits, as well as combining all of these into a single composite score of 
performance for the entire exercise’. In addition to the objective metrics the MdVT simulator 
defines thresholds which indicate whether the trainee’s score is considered a “passing” or 
“failing” performance with acceptable and warming scoring levels, respectively’. These 
thresholds were derived from data collection and analysis with a large number of experienced 
surgeons. Therefore, higher mental demand and frustration levels of trainees with simulation 
may suggest a bigger challenge and effort to accomplish the task sequentially through multiple 
repetitions of an exercise in order to reach the desirable “passing score”. 

Conversely, the animal hands-on part of the course did not have objective metric 
parameters to assess the skill set of trainees in robotics. The faculty of the course subjectively 
evaluated the proficiency levels of residents. Furthermore, the timeframe for every trainee with 
the robotic console was limited compared to the simulation part. 

These results combined with previous and future SES AUA training courses’ results can 
significantly enhance our efforts to establish a standardized robotic surgery training program that 
s cost effective, practical, and of the highest quality. Encouraging the development of urolo; 
specific robotic training tools in simulation will also aid in reaching our goal. Some limitations 
of this analysis include its regional focus and limited sample size. The analysis also did not 












































assess the methods each program uses for robotic training. Upon completion of residency 
programs, many urologists appreciate the effort and learning curve associated with robotic 
surgery, and believe that training and proficiency in robotic surgery are necessary during 
residency’. Future direction for this project includes compiling detailed accounts of trainees’ 
exposures at their home institutes following the training. Such analysis combined with ongoing 
performance scores and trainees’ subjective opinions could lead to identifying the most effective 
methods of training. Work is currently in progress to improve the current robotic training 
methods. 


CONCLUSIONS 


Novice trainees experienced significant mental workload while performing tasks on both 
the simulator and the live animal model during the robotics course. NASA TLX scoring 
demonstrates that live animal models provide the same proficiency performance with less 
frustration, The simulation part of course remains more challenging task for trainees with more 
frustration and repetitive exercises to achieve the passing score. 
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Appendix 1. 


Robotic Simulator Questionnaire 


1 


What year urology resident are you? 

O Urol 

O Uro2 

O Uro-3 

O Uro-4 

OD Uro-5 

Does your training program own or have access to a robotics simulator? 
No 

Mimic Simulator 

Ross Simulator 

Mimic Backpack or console 

Other 

Have you been on the robotics console for an actual case? 

O Yes 

0 No 

Approximate the number of cases on which you have robotics console time 
Oo «25 

O 26-50 

O 51-100 

O >100 

How do you rate your robotic training during residency? 

O Poor 

O Fair 

O Average 

O Excellent 

In your experience, do you feel that the simulator replicates real life robotics? 
O Yes 

0 No 

Which drill did you find the most dif 
OO Peg board 

1 Ring Walk 

Cl Thread the rings 

1 Tubes 2 

If your program lacks a robotics simulator, do you think this device would be helpful in 
your program? 

O Yes 

0 No 


oooog 





sult? 


Appendix 2. 


NASA Task Load Index 


Hart and Staveland's NASA Task Load Index (TLX) method assesses 
work load on five 7-point scales. Increments of high, medium and low 
estimates for each point result in 21 gradations on the scales: 


























Wome Task Dale 
Mental Demand How mentally demanding was the task? 
A PAT EC a Dy 
Very Low Very High 


Physical Demand How physically demanding was the task? 


Used Re ep ye Wa pt Te te fa ee 


Very Low Very High 





Temporal Demand How hurriad or rushed was the pace of the task? 


Tet ee Toad Pee Dp Pi Is Me eT 


Very Low Very High 





Performance How successtul were you in accomplishing what 
you were asked todo? 


|e TS al ER? eal E20 Cl ed YE aC A 


Perfect Face 





Effort How hare did you have to work to accomplish 
your level of performance? 


YY CE FS ER EE 


ry Low Very High 








Frustration How insecure, discouraged, initated, stressed, 
and annoyed wereyou? 


FO Te 0c Oe eG 
very Low Very High 





